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Abstract 

Tha  problems  of  developing  computerized  systems,  particularly 
management  information  systems  (MIS),  are  discussed  frequently  in  the 
literature.  However,  there  is  little  agreement  on  the  reasons  for 
failure  or  the  important  factors  for  success.  One  study  using  survey 
data,  performed  by  D.  M.  Carter,  et,  al.,  in  1975  found  four  factors 
critical  to  MIS:  Planning  and  Control,  Expertise,  Attitudes  and  Involve- 
ment. For  several  reasons,  a new  study  appeared  desirable.  A new  sur- 
vey was  developed  and  administered  to  456  computer  specialists  at  three 
Air  Force  bases.  Analysis  of  the  data  found  the  survey  samples  of  this 
and  the  Carter  study  similar,  although  not  the  same  statistically, 
number  of  the  factors  Important  to  MIS  were  found  to  vary  with  System 
Size,  Difficulty  and  Contractor  Involvement.  Factor  analyses  of  the 
sample  and  two  subsets  each  revealed  three  factors:  User  Involvement; 
Capabilities  of  the  Project  Organization;  and  Planning  and  Control. 
Several  predictive  models  were  developed  for  system  success,  the  best  of 
which  explained  43%  of  the  variance  in  success  for  the  total  sample,  and 
52%  of  the  variance  in  more  strongly  contracted  efforts.  It  was  con- 
cluded that  six  factors  are  critical  to  MIS,  the  three  factor  analysis 
factors  plus:  Size  and  Difficulty;  Criteria  for  Continuing  the  Project; 
and  Test  Time.  A process  model,  consistent  with  the  data,  of  the  initial 
stages  of  a MIS  was  proposed. 
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A STUDY  OF 


CRITICAL  FACTORS  IN  THE  DEVELOPMENT  OF 
AIR  FORCE  COMPUTERIZED  MANAGEMENT  INFORMATION  SYSTEMS 


I.  Background 


Introduction 

Delays,  from  various  causes,  arose  In  the  progress  of 
the  work,  and  great  expenses  were  Incurred.  The  machine  was 
altogether  new  In  design  and  construction.  . . "It  Involved," 
to  quote  again  from  the  Report  of  the  Committee  of  the  Royal 
Society,  "the  necessity  of  constructing,  and  In  many  Instances 
Inventing,  tools  and  machinery  of  great  expense  and  complexity." 

. . . Similar  circumstances  will,  I apprehend,  always  attend 
and  prolong  the  period  of  bringing  to  perfection  Inventions 
which  have  no  parallel  In  the  previous  history.  . . . The 
necessary  science  and  skill  specially  acquired  In  executing 
such  works  must  also,  as  experience  Is  gained,  suggest  devia- 
tions from,  and  improvements  in,  the  original  plan  of  those 
works.  ...  From  whatever  causa,  however,  the  delays  and  expenses 
arose,  the  result  was  that  the  government  was  discouraged  and 
declined  to  proceed  further  with  the  work  (Ref  1:2305-6). 

The  above  quote  Is  an  excerpt  from  a letter  from  Charles  Babbage  to 
Lord  Derby,  written  June  8,  1852,  explaining  the  problems  encountered  In 
constructing  a "Difference  Engine,"  the  first  mechanical  calculator. 

Such  were  the  troubled  beginnings  of  what  Is  now  a major  Industry  world- 
wide, the  processing  of  data  and  information  by  machine.  The  Industry 
has  undergone  tremendous  changes  since  Babbage.  Inexpensive  handheld 
calculators  now  perform  calculations  far  more  sophisticated  than  Babbage 
would  have  dreamed  possible.  Manufacturing  organisations  now  spend  more 
money  on  Information  than  on  direct  labor,  according  to  one  expert 
(Ref  2:257).  The  problems  which  frequently  accompany  large  projects. 
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however,  have  apparently  undergone  little  change.  Frederick  Brooks, 

for  example,  compares  large  scale  software  projects  to  the  tar  pits 

which  ensnared  dinosaurs  millions  of  years  ago  (Ref  3:4): 

Large-system  programming  has  over  the  past  decade  been 
such  a tar  pit,  and  many  a great  and  powerful  beast  have 
thrashed  violently  In  it.  Most  have  emerged  with  running 
systems— few  have  met  goals,  schedules,  or  budgets.  Large 
and  small,  massive  or  wiry,  team  after  team  has  become 
entangled  In  the  tar.  No  one  thing  seems  to  cause  the  diffi- 
culty—any  particular  paw  can  be  pulled  away.  But  the  accumu- 
lation of  simultaneous  and  Interacting  factors  brings  slower 
and  slower  motion. 

This  paper  seeks  to  investigate  the  problems  Involved  In  developing 
one  particular  class  of  computer  systems,  computerized  management  in- 
formation systems.  Specifically,  the  objectives  cf  this  study  are: 

1.  To  determine  what  factors  are  most  Important  to  the  successful 
development  of  Air  Force  computerized  management  Information 
systems, 

2.  To  determine  if  and  how  the  Important  factors  vary  with  a . 
number  of  system  and  project  attributes, 

3.  To  attempt  to  develop  a predictive  mooel  which  may  be  of  some 
use  In  evaluating  and  managing  future  development  efforts. 

In  pursuit  of  those  objectives,  this  chapter  discusses  the  nature  of 
management  Information  systems  (MIS),  research  into  critical  factors  for 
developing  MIS,  and  summarises  the  literature  on  Important  factors  for 
MIS.  The  second  chapter  describes  the  methodology  that  was  followed  In 
surveying  cos^uter  systems  personnel  at  three  Air  Force  Installations, 
and  In  analyzing  the  results  of  the  survey.  The  third  chapter  presents 
the  results  of  the  analysis,  and  the  final  chapter,  conclusions  and 
recommendations  for  further  research. 
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lng  management  information  systems  from  those  involved  in  other  kinds 
of  systems  for  a number  of  years.  One  of  the  central  problems  involved 
in  such  an  approach  is  that  there  is  no  single,  common  definition  for 
MIS.  Most  writers  avoid  defining  MIS  at  all.  The  definitions  available 
range  from  purely  on-line  Interactive  computer  systems  that  provide 
information  only  on  request  (Ref  4:54)  to  the  notes  that  the  owner  of  a 
small  business  might  carry  in  his  hat  (Ref  5:2).  Clearly  the  two  extremes 
involve  completely  different  problems.  Based  upon  research  used  in  de- 
veloping this  paper,  this  writer  contends  that  the  broader  definitions 
are  closest  to  what  practitioners  think  of  when  they  hear  the  term  MIS. 

For  purposes  of  this  study,  a management  information  system  is  de- 
fined as:  the  total  collection  of  resources  and  procedures  used  to 
collect,  process  and  disseminate  the  information  used  by  managers  to  make 
the  decisions  required  to  further  the  objectives  of  the  organisation. 

Given  this  definition  it  is  apparent  that  any  organisation  has  such  a 
system.  It  is  also  apparent  that  any  such  system  also  has  two  basic 
parts:  a formally  sanctioned  or  "official"  part  and  an  informal  part. 

Many  organisations  have  in  some  form  or  another,  taken  a further  subset 
of  the  formal  Information  system  and  automated  it.  This  is  the  computer- 
ised management  information  system.  For  purposes  of  this  study,  a 
computerised  management  information  system  is  defined  as:  a computer 
system  which  is  used  as  part  of  the  total  collection  of  resources  and 
procedures  used  to  collect,  process  and  disseminate  the  information  used 
by  managers  to  make  the  decisions  required  to  further  organisational 
objectives. 
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What  Is  it  then,  that  makes  developing  a computerized  MIS  different 
from  developing  any  other  computerized  system?  I contend  that  the  main 
differences  arise  from  the  fundamental  nature  of  MIS  in  general  and  to  a 
lesser  extant  to  the  inherent  difficulties  involved  in  any  computer  system 
which  is  used  by  non-ccaputer  specialists. 

In  any  MIS  where  personnel  other  than  the  using  managers  are  made 
responsible  for  collecting  or  reporting  information,  it  is  necessary  to 
identify  some  form  of  an  Information  requirement.  This  is  a fairly 
trivial  step  for  the  small  businessman  mentioned  earlier,  but  more  diffi- 
cult in  more  complex  organizations.  It  requires  that  the  responsibilities 
of  related  departments  be  somehow  delineated.  It  requires  that  the  de- 
cisions required  by  each  be  identified.  The  Information  which  is  needed 
to  support  those  decisions  must  then  be  Identified  in  terms  of  level  of 
aggregation,  timing,  and  currency.  This  is  not  possible  if  the  managers 
cannot  identify  what  information  is  needed  to  make  the  decisions,  if  the 
problem  is  not  easily  solved  by  analytical  techniques. 

The  difficulties  of  MIS  become  compounded  in  any  organization  which 
operates  in  a changing  environment,  is  characterized  by  frequent  internal 
organizational  changes,  or  in  which  some  departments  make  decisions  re- 
quiring information  which  is  the  "property"  of  other  departments. 

A changing  environment  demands  different  decisions  which  requires  chat 
the  cycle  of  identifying  the  information  and  defining  the  requirement 
must  be  repeated.  Internal  organizational  changes  create  confusion  over 
roles  and  responsibilities  making  it  difficult  to  determine  "who”  makes 
"what"  decisions.  When  information  is  shared  or  passed  between  depart- 
ments, it  is  necessary  to  establish  the  authority  and  the  priority  of  the 
using  department  to  have  the  information. 
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MIS  are  also  often  beset  by  any  number  of  human  behavioral  problems, 
especially  when  the  system  is  used  for  performance  evaluation  and  control. 
It  is  frequently  the  case  in  such  systems  that  the  personnel  whose  activi- 
ties are  to  be  controlled  with  the  system  are  the  same  personnel  required 
to  prepare  Inputs  to  the  system.  In  these  cases  it  is  very  difficult  to 
establish  the  motivation  necessary  to  Insure  accuracy,  and  the  system  may 
even  be  "sabotaged"  with  Inaccurate  Inputs.  In  the  case  of  using  a system 
for  performance  evaluation,  it  is  very  difficult  to  develop  performance 
measures  which  cannot  be  "gamed"  nor  result  in  dysfunctional  behavior. 

That  is,  to  the  extent  that  the  performance  measure  is  a substitute  for 
organizational  objectives,  the  individual's  or  the  department's  goal 
becomes  the  substitute  measure  rather  than  the  real  organizational  goal. 

Given  the  problems  Inherent  to  MIS  in  general  computerized  MIS  are 
subject  to  additional  problems  due  to  their  use  by  non-computer  special- 
ists. Among  these  are  communication  between  user  and  computer  personnel 
and  resistance  to  automation  and  change.  As  indicated  earlier,  any  time 
the  information  needed  to  make  a decision  is  to  be  collected  by  other  than 
the  using  manager,  it  is  necessary  to  develop  some  form  of  an  Information 
requirement.  This  problem  is  compounded  in  computerized  systems  for  at 
least  two  reasons,  first,  the  using  manager  probably  does  not  understand 
what  the  computer  can  and  cannot  do  for  him  and  therefore  does  not  knov 
what  to  ask  for.  Second,  the  computer  specialist  probably  does  not 
understand  what  the  user  needs  and  therefore  what  to  give  him. 

The  problems  of  resistance  to  change  and  automation  have  received  a 
great  deal  of  attention  in  the  literature.  However,  these  problems  are 
probably  greatest  for  MIS  because  of  the  closer  interaction  with  non- 
computer personnel  than  In  most  other  computer  systems,  and  because  of 
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the  use  of  the  system  as  a medium  for  control  and  performance  measure- 
ment, as  discussed  earlier. 


The  differences  between  computerized  management  information  systems 
and  other  computerized  systems  can  be  summarized  as  follows: 

1.  Management  information  systems  have  as  an  objective  providing 
information  to  managers  for  decision-making,  but  the  decisions 
and  the  information  required  to  support  che  decisions  are 
often  difficult  to  identify. 

2.  The  decisions  made  by  managers  may  easily  be  changed  by  changes 
in  either  the  external  or  internal  organizational  environment. 
Thus  the  MIS  is  in  turn  easily  affected  by  organizational 
changes.  These  two  effects  interact  with  each  other. 

3.  MIS  frequently  require  inputs  from  departments  other  than  the 
departments  using  the  information.  This  creates  organizational 
conflict  and  lack  of  support  by  the  "input”  organizations. 

4.  MIS  are  often  used  for  performance  measurement  and  control. 

is  may  result  in  inaccurate  inputs. 

3.  Computerized  MIS  depend  upon  the  ability  of  user  managers  and 
computer  specialists  to  communicate  very  effectively  to  over- 
come the  lack  of  a common  background. 

6.  Computerized  MIS  are  probably  more  affected  by  resistance  to 
change  and  automation  than  are  other  systems. 

Of  the  differences  listed,  the  MIS  objective  of  assisting  and  sup- 
porting the  decision  making  activities  of  managers  has  probably  received 
the  most  attention  in  the  literature.  Perhaps  this  is  due  in  part  to  the 
Inherent  difficulties  involved  in  defining  and  quantifying  the  Inputs  and 
parameters  to  many  management  decisions.  Were  all  such  decisions  capable 
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of  complete  definition,  automation  would  probably  be  possible  and 
managers  unnecessary.  In  any  case,  it  is  those  undefinable  decisions 
which  may  be  most  important  to  the  continued  prosperity  of  the  organiza- 
tion. Cne  example  can  be  found  in  the  conduct  of  war,  which  few  would 
argue  could  ever  be  completely  automated.  On  that  subject,  former 
Deputy  Secretary  of  Defense  William  P.  Clements,  Jr.  stated,  "Throughout 
history,  the  outcome  of  conflict  has  been  determined  as  much  by  the  col- 
lection and  proper  use  of  good  Information  as  it  has  been  by  the  quality 
and  quantity  of  weaponry"  (Ref  6:10). 

Background  of  Critical  Factors  Research 

The  Inadequacies  in  computer  system  development  projects  have  been 

stated  time  and  time  again  in  the  literature: 

Many  organizations  have  experienced  serious  difficulties 
in  developing  complex  computer-based  systems,  especially  their 
software  components . The  problems  Include  large  cost  overruns, 
schedule  slippages,  inadequate  performance,  and  Inability  to 
use  the  system  as  envisaged  (Ref  7:1). 

Both  surveys  and  practical  experience  have  indicated  over- 
whelmingly that  most  computer  systems  Installed  during  the 
fifties  and  sixties  have  failed  in  at  least  some  Important 
respect  (Ref  8:3). 

. . . software  is  often  excessively  costly,  late  in  com- 
pletion, poor  in  performance,  and  "unreliable"  (Ref  9:296). 

As  indicated  earlier,  the  lack  of  any  uniform  definition  of  the 

term  MIS,  makes  it  difficult  to  determine  specifically  what  the  problems 

of  developing  MIS  are.  One  result  of  this  situation  la  that  when  vlevlng 

quotes  such  as  those  listed  above,  it  is  impossible  to  determine  to  what 

extant  the  problems  in  the  past  resulted  from  MIS,  and  to  what  extent 

the  problems  resulted  from  other  systems.  The  problems  of  MIS  have 
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received  specific  mention  on  a number  of  occasions,  however: 

Abandoning  multimillion  dollar  MIS  efforts  is  a relatively 
common  place  occurrence  today  (S.ef  10:29). 

Management  information  system  projects  generally  have 
two  distinguishing  characteristics:  1)  they  are  late,  and  2) 
there  is  usually  a significant  cost  overrun  (Ref  11:73). 

...  most  Information  systems  have  failed  . . . (Ref  12:6). 

Not  all  of  the  authors  agree  on  the  magnitude  of  the  problem.  Long 
contends  that  while  the  problems  have  been  widely  publicised,  some  firms 
have  successfully  been  developing  MIS  quietly  under  wraps  (Ref  13:23). 

The  Air  Force  however  had  a very  visible  example  of  failure  with  the 
cancellation  of  the  $800  million  Advanced  Logistics  System  in  1973  by 
Congress  (Ref  14:68).  The  problems  of  that  system,  as  reported  In  the 
ALS  Assessment,  are  in  my  opinion  remarkably  similar  to  the  problems 
encountered  by  Babbage  and  his  "Difference  Engine"  cited  at  the  begin- 
ning of  this  study,  and  of  course  both  systems  met  the  same  fate  at  the 
hands  of  the  legislature.  Although  the  degree  of  schedule  slippage 
cannot  be  determined  from  the  ALS  Assessment,  the  initial  contract  cost 
at  award  was  to  be  $80  million  over  the  life  of  the  system  (Ref  13:22). 
This  cost  cannot  be  compared  with  the  cost  at  cancellation,  because  that 
higher  cost  included  substantial  amounts  for  in-house  programming  efforts. 
However,  it  does  give  at  least  a sense  of  the  probable  magnitude  of  the 
overrun. 

Despite  general  agreement  that  MIS  projects  have  been  less  than 
completely  successful,  there  is  very  little  agreement  concerning  the 
reasons  for  failure  and  the  most  important  factors  for  success.  A few 
authors  have  cited  a single  most  Important  reason.  Axelson  states  that 
success  depends  on  all  levels  of  the  organisation  recognizing  the 


Importance  of  the  system  and  that  most  failures  were  due  to  lack  of 
management  involvement  In  the  project  (Ref  15:26).  Lucas  says  the  most 
Important  reason  for  failure  is  that  "...  we  have  ignored  organizational 
behavior  problems  in  the  design  and  operation  of  computer-based  informa- 
tion systems"  (Ref  12:6).  Turn  says  "One  major  reason  for  such  lack  of 
success  has  been  the  inability  of  management  of  the  organization  or  the 
development  effort  to  understand  the  need  for  a total-system  management 
approach"  (Ref  7:1).  Oonelson  says  "With  better  project  management  tools, 
virtually  all  DP  [Data  Processing]  system  Implementations  can  be  success- 
ful and  cost  effective"  (Ref  11:73).  Brooks  says  "More  software  projects 
have  gone  awry  for  lack  of  calendar  time  than  for  all  other  causes 
combined"  (Ref  3:14).  The  important  factors  and  relative  lack  of  agree- 
ment are  more  fully  discussed  in  the  next  section.  However,  at  this 
point  the  following  conclusions  can  be  drawn: 

1.  Successful  management  of  MIS  projects  is  a complex  and 
little  understood  subject. 

2.  There  is  a need  for  empirical  research  into  what  factors  are 
most  important  to  the  successful  development  of  MIS,  so  that 
future  projects  can  be  enhanced  by  concentrating  on  the  most 
important  Issues. 

One  such  research  effort  vas  undertaken  by  a team  of  researchers 
at  Colorado  State  University  (CSU),  led  by  0.  M.  Carter.  The  effort  was 
performed  under  contract  with  the  Air  Force  Office  of  Scientific  Research 
during  the  period  September,  1972,  to  March,  1975  (Ref  17:1-5).  The 
purpose  of  the  CSU  study  can  be  summarized  (Ref  17:1): 

The  effectiveness  of  computer-based  information  systems 

undoubtedly  is  dependent  upon  many  factors.  Factors  usually 


9 


discussed  are S hardware,  systems  personnel,  operating  manage* 
sent  ...  and  so  forth,  to  an  almost  Inexhaustible  list.  A 
major  question,  however,  has  not  been  answered:  what  are  the 
most  critical  factors  that  contribute  to  the  successful  de- 
velopment  of  an  Information  system?  In  addition,  can  the 
factors  be  measured?  If  an  organization  could  measure  the 
value  of  each  factor  and  obtain  a statistical  composite  value 
of  all  factors,  predictions  could  be  made  as  to  the  probable 
success  or  failure  of  a proposed  system  effort. 

the  methodology  followed  by  the  CSU  researchers  was  to  first  de- 
termine what  factors  were  potentially  most  important,  and  then  to  reduce 
these  factors  to  the  critical  factors.  This  was  followed  by  efforts  to 
develop  survey  and  other  measuring  instruments  to  determine  the  level  of 
attainment  of  each  factor  in  a given  systems  effort.  The  researchers 
then  developed  an  interactive  goal  programming  model  to  assist  project 
personnel  in  determining  what  variables  and  factors  needed  increased 
attention  in  order  to  achieve  project  success  (Ref  17). 

In  order  to  determine  which  factors  were  important,  the  CSU  re- 
searchers first  interviewed  a total  of  40  systems  and  general  management 
personnel  in  business  and  government  agencies  in  Colorado.  Based  upon 
the  interviews,  a 20  factor  checklist  was  developed  and  tested  on  the 
same  personnel.  Based  upon  these  results,  a refined  14  factor  checklist 
was  sent  to  200  systems  analyst  and  management  personnel  throughout  the 
U.  S.  The  researchers  received  120  usable  returns  and  the  results  are 
shown  in  Table  I. 

The  researchers  performed  an  analysis  of  the  ten  factors  common  to 
the  two  checklists  using  the  Spearman  rho  statistic  to  verify  that  the 
rankings  were  from  the  same  population.  Additionally,  analysis  of 
variance  was  used  to  verify  that  the  means  of  the  factor  ratings  did  not 
vary  significantly  by  industry  type.  The  researchers  then  used  factor 
analysis  to  group  the  14  factors  into  four  ncritlcaln  factors.  These 
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Table  I 

Results  of  the  CSU  Critical  Factors  Checklist 


Mean 

Std. 

Dev. 

Factor 

1 

.89 

.15 

Definition  of  the  objective  of  a specific 
information  system 

2 

.88 

.13 

Identification  of  the  information  needs  of 
management 

3 

.81 

.17 

Attitude  of  management  toward  a given  system 
effort 

4 

.80 

.16 

Communication  abilities  of  design  team 

5 

.72 

.20 

Expertise  and  creative  ability  of  design  team 

6 

.72 

.30 

Determination  of  company  objectives 

7 

.68 

.27 

Justif i cat ion  of  system  cost 

8 

.66 

.29 

Participation  of  management  on  design  team 

9 

.65 

.23 

Adequacy  of  time  frame  for  system  effort 

10 

.64 

.22 

Attitudes  of  design  team  toward  user  department 

11 

.61 

.26 

Attitudes  of  user  department  toward  design  team 

12 

.60 

.25 

Resistance  to  change  by  management 

13 

.30 

.23 

Resistance  to  change  by  employees 

14 

.47 

.26 

Sophistication  and  ambitiousness  of  project 

(From  Ref  17:10) 


factors  were  named  planning  and  control , attitude,  expertise  and  involve* 
aont.  The  researchers  then  use  "selected  systems  educators  and  prac- 
tit loners"  to  weight  each  of  the  factors  in  terms  of  Importance  to 
project  success  (Ref  17:7-16). 

The  procedures  used  by  the  CSU  researchers  in  determining  the  im- 
portant factors  were  critical  to  all  of  the  subsequent  work  performed  by 
the  team.  The  use  of  factor  analysis  to  determine  the  "critical"  factors 
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was  an  abstraction  from  the  original  data.  The  efforts  to  determine 


appropriate  weights  for  the  critical  factors  and  ways  to  measure  the 
level  of  attainment  of  those  factors  were  both  further  abstractions. 

Thus  the  accuracy  of  the  original  data  was  critical  if  the  subsequent 
work  was  to  be  usable.  However,  the  writer  found  a number  of  potential 
problems  with  the  original  data,  such  that  a further  survey  effort  ap- 
peared desirable.  These  problems  are: 

1.  Although  the  research  was  to  determine  factors  critical  to 
the  development  of  Air  Force  systems,  only  non- Air  Force 
agencies  were  surveyed. 

2.  The  survey  asked  each  respondent  to  rate  each  of  the  factors 
for  systems  in  general. 

3.  The  survey  gave  no  indication  of  the  appropriate  level  of 
attention  that  should  be  given  each  factor. 

A.  The  survey  did  not  appear  sufficiently  inclusive. 

With  respect  to  the  first  point,  the  CSU  researchers  surveyed  non- 
Air  Force  institutions  because  of  the  accessibility  of  those  institutions. 
The  researchers  "believed  a high  degree  of  correlation  to  exist  between 
critical  factors  in  Air  Force  systems  and  factors  in  non-Alr  Force 
systems"  (Ref  17:7).  Uiis  hypothesis  was  at  least  partly  supported  both 
by  findings  that  the  two  survey  groups  rated  the  factors  the  same  in  a 
statistical  sense,  and  Chat  the  means  of  the  factor  ratings  did  not  vary 
significantly  with  industry  type  (Ref  17:11-12). 

However,  this  researcher  feels  that  there  are  a number  of  reasons 
why  critical  factors  in  the  Air  Force  may  be  significantly  different. 

Most  authors,  for  example,  place  considerable  emphasis  on  determining 
the  objectives  of  the  organisation.  This  process  is  complicated  in  the 


12 


Air  Fore*  by  at  lease  three  Issues:  the  fact  that  the  Air  Force  is  a 
government  organisation  and  must  satisfy  a multitude  of  goals  beyond  the 
basic  mission;  the  absence  of  an  easily  definable  and  measurable  product 
(combat  capability);  and  the  absence  of  a profit  motive.  Other  potential 
reasons  why  the  Air  Force  may  be  different  include  the  size,  nature  and 
intended  use  of  Air  Force  systems,  as  well  as  the  complex  organizational 
relationships  that  frequently  occur  in  the  Air  Force.  The  last  point 
will  be  discussed  in  the  next  chapter.  With  respect  to  the  other  Issues, 
the  Air  Force  has  been  Involved  in  a number  of  extremely  large  projects, 
such  as  the  ALS  and  the  Worldwide  Military  Command  and  Control  System. 

Air  Force  projects  since  the  Seal-Automatic  Ground  Environment  system 
have  frequently  been  on  the  forefront  of  the  state-of-the-art.  Also  in 
command  and  control  systems,  for  example,  speed,  reliability  and  flexi- 
bility become  far  more  critical  than  in  most  business-oriented  systems. 
Therefore,  1 contend  that  it  may  be  very  useful  to  know  not  only  if  the 
factors  vary  for  Air  Force  versus  non -Air  Force  systems , but  also  whether 
the  important  factors  vary  significantly  across  a number  of  other  param- 
eters. These  include  the  Intended  use,  size,  sophistication,  project 
organization,  maintenance  arrangements  and  contractual  arrangements. 

The  problem  with  asking  survey  respondents  to  rate  the  factors  in 
terms  of  importance  to  systems  efforts  in  general  is  that  there  may  be  a 
tendency  to  rat*  according  to  some  theory  or  teaching,  rather  than  by 
experience.  The  opposite  problem  nay  be  encountered  by  those  respondents 
who  attempt  to  rely  on  experience.  The  factor  checklist  asked  the  re- 
spondents to  rat*  each  factor  in  terms  of  importance  "...  to  the  systems 
effort”  (Ref  17:82).  The  problem  is  that  importance  to  success  does  not 
necessarily  equate  with  level  of  effort  or  attention.  For  example,  a 
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factor  may  be  critically  important  if  completely  ignored,  yet  only 
slightly  important  if  given  minimal  attention.  The  opposite  situation 
could  also  occur,  if  a factor  was  accorded  too  much  attention.  Thus,  it 
may  be  important  and  useful  to  both  tie  the  ratings  to  specific  develop- 
ment efforts,  and  to  achieve  some  sense  of  the  appropriate  level  of 
effort  for  each  factor. 

II 

The  final  point,  that  the  critical  factor  checklist  was  not  suf- 
ficiently inclusive,  will  become  more  evident  In  the  next  section.  In 
summary  the  writer  believes  that,  while  the  CSU  study  was  enlightening 
and  a valuable  effort,  trtft  a new  survey  is  needed  to  overcome  some  of 
the  potential  difficulties  of  the  CSU  critical  factor  checklist.  The 
survey  instrument  developed  by  this  researcher  (discussed  in  the  next 
chapter)  attempts  to  overcome  these  difficulties. 


Literature  Sear 


As  a first  step  in  this  effort,  a literature  search  was  performed 
in  order  to  obtain  an  improved  list  of  factors  important  to  the  success- 


ful development  of  MIS.  The  search  was  broadly  based,  both  because  of 
the  lack  of  a uniform  definition  of  MIS  and  the  fact  that  MIS  development 
inherently  involves  problems  of  both  software  development  and  project 
management.  Relevant  articles  and  reports  were  found  under  such  classi- 
fications as:  automation,  computers,  electronic  data  processing, 
management,  management  systems,  planning  and  control,  project  management 
and  software  development.  A total  of  over  ISO  significantly  different 
and  potentially  Important  "factors"  or  "problems"  were  listed  by  the 
more  than  SO  references  surveyed.  Despite  the  numbers  Involved,  the 
writer  was  only  reasonably  confident  that  the  full  breadth  of  opinion 
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had  been  determined.  As  an  Indication  of  the  diversity  of  opinion,  at 
least  eight  distinct  and  well-developed  models  of  the  steps  Involved  in 
developing  MIS  or  software  were  located  (Refs  7,8,19,20,21,22,23). 

As  shown  earlier,  a few  authors  did  list  a single  most  important 
factor.  In  addition,  several  listed  a relatively  manageable  number  of 
important  factors.  A few  of  these  are  shown  in  Table  II.  This  table 
also  gives  a little  better  feel  for  the  possible  existence  of  some  truly 
common  factors.  These  might  include  management  involvement  and  user 
needs.  By  looking  at  the  important  factors  listed  by  the  30  plus  authors, 
the  writer  was  able  to  pick  out  some  candidate  critical  factors,  simply 
by  finding  the  factors  listed  most  often.  These  were  (followed  by  the 
number  of  authors  specifically  listing  the  factor): 

Management  involvement  (13) 

User  involvement  (13) 

Some  form  of  disciplined  project  management  approach, 
characterized  by  phased  reviews  (9) 

Some  form  of  formalized  change  control  or  configuration 
management  (8) 

Information  needs  of  users  (7) 

("Total"  needs  also  listed  by  2) 

Human  factors  (5) 

System  objectives  (3) 

Adequacy  of  planning  (4) 

System  requirements  determination  (4) 

("Complete,  unambiguous,  testable"  requirements  also 
listed  by  3). 

A variety  of  project  management  approaches  were  proposed.  Of  these, 
however,  nine  authors  specifically  stressed  the  need  for  periodic  manage- 
ment reviews  (generally  including  user  representatives)  during  the  project. 
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Table  II 

Factors  Listed  by  Selected  Experts 


Joseph  Orlicky  (Ref  4:4): 

1.  Understanding  of  the  business  function  to  be  automated. . . 

2.  Motivation  to  innovate. 

3.  Competence  in  design  and  ability  to  relate  the  system's 
functions  to  operating  needs. 

4.  Authority  and  prestige  (of  project  personnel). 

3.  Insight  into  how  human  factors  affect  the  success  of  a 
computer-based  system. 

John  V.  Soden  (reasons  for  failure)  (Ref  10:29): 

1.  User  needs  not  fully  understood  before  systems  design. 

2.  Alternative  system  designs  not  evaluated  for  costs  and 
benefits. 

3.  Performance  due  to  technical  design. 

4.  Capability  to  perform  within  budget. 

3.  Scheduling  of  potential  MIS  not  based  upon  organizational 
objectives. 

Joel  E.  Ross  (reasons  for  failure)  (Ref  24:36): 

1.  Lack  of  management  involvement. 

2.  Too  much  useless  data. 

3.  The  communications  gap  (between  users  and  systems). 

4.  Lack  of  planning. 

3.  Failure  to  Identify  Information  needs. 

6.  Reliance  on  manufacturer  or  consultant. 

Martin  J.  Shio  (reasons  for  failure)  (Ref  23:38-9): 

1.  Lack  of  proper  objectives  and  goals  for  many  MIS  projects. 

2.  Lack  of  management  participation  and  development. 

3.  Control  measures  and  evaluative  criteria  are  inadequate 
or  absent. 

4.  False  assumptions  are  used  in  designing  MIS. 

3.  Inability  of  many  system  designers  to  Identify  information 
needs  for  managers. 

6.  Lack  of  flexibility  in  many  MIS. 

7.  Poor  implementation  plan. 

8.  Too  much  emphasis  is  placed  on  technical  aspects  while 
placing  relatively  little  emphasis  on  human  factors. 
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Information  needs  of  users  was  listed  by  seven  authors,  and  an  additional 
two  felt  it  important  to  stress  the  "total”  needs  of  users.  System  re* 
quirements  determination  (or  phase)  was  listed  by  four,  and  three  others 
indicated  the  attributes  of  requirements  shown. 

The  critical  nature  of  user  Involvement  is  further  demonstrated  by 
Table  III,  which  consists  of  a partial  list  of  other  factors  Hsted  by 
the  authors  relating  directly  to  user  involvement.  As  can  be  seen  from 
the  list,  user  involvement  Is  far  more  important  than  the  13  "votes" 
would  indicate;  furthermore,  the  authors  disagree  substantially  on  the 
best  way  to  achieve  user  Involvement.  The  critical  nature  of  user 
involvement  was  particularly  well-demonstrated  by  the  results  of  a study 
of  56  systems  performed  by  Alter  (Ref  27:103) : 

Intended  users  neither  initiated  nor  played  an  active 
role  in  implementing  11  of  the  15  systems  that  suffered  sig- 
nificant implementation  problems.  Conversely,  there  were 
relatively  few  such  problems  in  27  of  the  31  systems  in  which 
users  had  a hand  in  initiating  and/or  played  an  active  role  in 
implementing. 

Although  management  involvement  also  received  13  "votes,"  far  fewer 
factors  were  specifically  related  to  management  Involvement  than  were  to 
user  involvement.  Perhaps  this  is  due  in  part  to  some  conceptual  diffi- 
culty on  the  part  of  some  authors  in  differentiating  users  from  managers. 
Generally  speaking,  the  writer  found  the  issue  clearest  when  "managers" 
was  used  to  refer  only  to  "top  management.”  The  factors  relating  to 
management  involvement  were  (not  direct  quotes) : 

Management  knowledge  and  experience  with  computers  (Refs  4,7) 

Task  force  of  top  management  used  in  problem  definition  (Ref  32) 

Top  management  control  of  system  development  efforts  (Refs  8,13,34) 

Removal  of  the  human  and  organizational  barriers  to  system 
development  (Ref  5) 
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Table  III 

Factors  Relating  to  User  Involvement* 


1.  Quality  of  user  team  members  (Refs  4,8). 

2.  User  experience  and  knowledge  of  computers  (Rofs  4,21,25). 

3.  Promoting  the  system  to  users  (Refs  8,27,28). 

4.  Users  perform  testing  (Refs  7,8). 

3.  Continuing  MIS  education  of  users  (Ref  13). 

6.  Project  managed  by  users  (Refs  29,30). 

7.  System  goals  same  as  user  goals  (Ref  29). 

8.  Final  decision  belongs  to  the  user  (Refs  8,26,29). 

9.  User  responsibility  of  project  success  (Ref  27). 

10.  Communications  between  users  and  systems  personnel  (Refs  7,24). 

11.  Systems  personnel  knowledge  of  user  goals  and  operations 
(Refs  26,31). 

12.  User  control  of  requirements,  but  with  the  use  of  an  independent 
validation  agency  (Ref  7). 

13.  Continuous  evaluation  (during  operation)  of  user  satisfaction 
with  the  system  (Ref  28). 

14.  User  training  begun  early,  performed  intensively  (Refs  4,8,32). 

15.  User  should  participate,  but  design  and  develop  Independently 
to  control  changes  (Ref  7). 

16.  Full-time  systems  representatives  in  user  departments  (Ref  29). 

17.  Training  of  employees  performed  by  users  (Refs  4,8). 

18.  Total  needs  of  users  (Refs  10,28). 

19.  Information  needs  of  users  (Refs  4,8,24,25,29,33,34). 

20.  User  commitment  to  achieving  the  benefits  of  the  system 
(Refs  4,8). 

21.  User  acceptance  and  commitment  to  Implementing  the  operation.  1 
changes  required  by  the  system  (Refs  8,21,34). 

22.  Evaluation  of  the  impact  of  the  system  on  users  (Refs  8,21). 

23.  Identification  of  organisational  responsibilities  and  support 
tasks  that  will  be  required  (Refs  29,34). 

24.  Systems  personnel  respect  for  the  users  as  "owners"  of  the 
system  (Ref  8). 

25.  User  participation  in  project  reviews  (Refs  7,35,36). 

*Factors  are  not  direct  quotes. 


Treating  Information  as  a major  business  function  (Ref  16) 

Ability  of  top  management  to  evaluate  the  system  (Ref  33) 

Direction  from  top  management  on  development  policy  and  procedural 
changes  (Ref  32). 
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Given  Che  complex,  contradictory  and  apparently  unlimited  advice 
in  the  literature,  the  question  posed  by  the  CSU  researchers  remains: 

",  . . what  are  the  most  critical  factors  that  contribute  to  the  suc- 
cessful development  of  an  Information  system"  (Ref  17:1)?  If  the  CSU 
researchers  were  correct,  there  should  be  a relatively  manageable  number 
of  truly  common  and  critical  factors.  Shaw  tends  to  agree  when  he  says 
(Ref  8:26): 

Experience,  then,  has  proved  that  there  are  certain  common 
denominators  involved  in  planning,  developing,  and  implementing 
a computerized  data  processing  system.  These  common  denominators, 
as  far  as  can  be  determined,  apply  to  virtually  all  system 
development  situations. 

Although  Shaw  does  not  distinguish  MIS  from  other  systems,  in  my  opinion 
the  development  process  propounded  by  Shaw  appears  specifically  geared 
tovard  MIS.  Regardless  of  whether  the  factors  are  common  to  MIS  or  all 
systems,  the  sheer  number  of  factors  alleged  to  be  applicable  to  MIS 
presents  a real  problem  in  terms  of  developing  a methodology  for  deter- 
mining the  truly  critical  factors.  Obviously  it  would  be  impractical  to 
design  an  effective  survey  for  measuring  ISO  factors. 

As  an  initial  step  in  reducing  the  number  of  factors  to  a manageable 
number,  the  writer  developed  the  list  shown  below.  This  list  is  a 
synthesis  of  the  more  prevalent  views  in  the  literature.  Further  re- 
finements were  made  in  developing  the  survey  and  are  discussed  in  the 
next  chapter.  The  categories  under  which  each  of  the  factors  is  listed 
are  not  intended  as  steps  in  a development  process,  but  rather  as  a class 
of  activities.  Even  this  distinction  is  somewhat  arbitrary,  since  many 
of  the  factors  apply  to  several  activities. 
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Environmental  Factors 


1.  Information  treated  as  a major  function. 

2.  Formalized  organizational  goals. 

3.  Use  of  long-range  organizational  planning. 

4.  Use  of  long-range  systems  planning  to  support 
organizational  plans. 

3.  Management  and  user  knowledge/experiance  with  computers. 

6.  Attitude  of  users  toward  systems  personnel/projects. 

7.  Attitude  of  systems  personnel  toward  users. 

8.  Involvement  and  support  by  top  management  in  system  development. 
Project  Management  Factors 

1.  Quality  of  project  managers. 

2.  Use  of  a disciplined  development  approach  with  periodic 
reviews  Involving  users  and  top  management. 

3.  Emphasis  on  the  planning  aspects  of  projects. 

4.  Use  of  formalized  change  control  procedures. 

3.  Commitment  by  users  to  the  success  of  the  project. 

6.  Management  control  system  (e.g.t  PERT). 

Project  Initiation  Factors 

1.  Identification  of  system  requirements. 

2.  Identification  of  system  performance  and  acceptance  criteria. 

3.  Identification  of  project  success  criteria. 

4.  Identification  of  milestones. 

3.  Definition  of  the  size  and  scope  of  the  project. 

6.  Identification  of  system  objectives. 

7.  Justification  of  system  cost. 

8.  Relationship  between  system  goals  and  long-range  plans. 

9.  Prioritized  development  of  systems. 

10.  Integration  of  systems. 

11.  Identification  and  documentation  of  the  assumptions  of 
the  project. 

12.  Adequacy  of  calendar  time  for  the  project. 

13.  Assignment  of  adequate  resources  to  the  project. 

14.  Identification  and  adequacy  of  the  authority  and  responsibility 
of  the  project  manager. 

13.  Use  of  a project  organization. 

Analysis  Factors 

1.  Objectives  of  the  user  organization. 

2.  Analyst  knowledge  of  user  operations. 

3.  Evaluation  of  the  impact  of  the  system  on  users. 

4.  Information  needs  of  users. 

3.  Experience  and  creative  ability  of  analyst  personnel. 
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Design  Factors 

1.  Use  of  a disciplined  design  approach. 

2.  Thoroughness  of  design. 

3.  Ability  to  view  the  system  as  a complete  entity. 

4.  Use  of  automated  software  development  tools. 

3.  Specification  of  run,  module  and  interface  requirements. 

6.  Abilities  of  design  personnel. 

7.  Continuity  of  work  force. 

8.  Evaluation  of  alternative  designs. 

9.  Quality,  timing  and  usability  of  output. 

10.  Simplicity  and  logic  of  the  system  design. 

11.  Responsiveness  of  the  system  to  special  requests. 

12.  Adequacy  of  hardware. 

13.  Staffing  of  the  design  team. 

14.  Management  and  user  Involvement  in  design. 

13.  Communication  abilities  of  the  design  team. 

16.  Use  of  a prototype. 

Programming  Factors 

1.  Use  of  a standard  high-order  language. 

2.  Modularity  of  programs. 

3.  Maintainability,  reliability  and  adaptability  of  programs. 

4.  Documentation. 

3.  Identification  and  control  of  work  assignments. 

6.  Use  of  methods  and  work  standards. 

Test  Factors 

1.  Adequacy  of  calendar  time  for  test. 

2.  Use  of  a formal  test  plan  and  methodology. 

3.  Availability  of  hardware  and  software  for  debugging  programs. 

4.  Adequacy  of  test  data. 

5.  User  Involvement  in  test. 

Implementation  Factors 

1.  Commitment  by  users  to  achieve  the  benefits  of  the  system. 

2.  Commitment  to  implement  the  procedural  and  organisational 
changes  required  by  the  system. 

3.  Education  and  training  of  users. 

4.  Use  of  a formalized  problem  reporting  system. 

3.  Promotion  of  the  system  to  users. 

6.  Formalized  turnover  or  transition. 

7.  Performance  of  a post-implementation  review. 

This  chapter  has  discussed  the  nature  of  management  information 
systems  and  a literature  review  of  the  research  addressing  the  factors 
critical  for  development  of  MIS.  The  second  chapter  describes  the 
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methodology  that  was  followed  in  surveying  computer  systems  personnel 
at  three  Air  Force  installations,  and  in  analyzing  the  results  of  the 
survey.  The  third  chapter  presents  the  results  of  the  analysis,  and 
the  final  chapter,  conclusions  and  recommendations  for  further  research. 
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II.  Methodology 


The  Survey  Instrument 

Ac  discussed  earlier,  the  survey  used  in  this  study  attempts  to 
overcame  four  potential  problems  with  the  CSU  14  factor  checklist. 

These  were: 

1.  Administering  the  survey  to  only  non- Air  Force  agencies. 

2.  Not  tieing  the  factor  ratings  to  specific  systems. 

3.  Not  obtaining  a measure  of  the  appropriate  level  of  attention 
that  should  be  given  each  factor. 

4.  Not  including  enough  factors  in  the  checklist. 

A copy  of  the  survey  developed  and  administered  for  this  study  is  shown 
in  Appendix  A.  The  survey  consists  of  three  parts.  Part  A measures  7 
demographic  variables.  Part  B 12  system  attributes  and  Part  C the  Import- 
ance of  40  different  factors.  The  first  problem  was  overcome  simply  by 
giving  the  survey  within  Air  Force  organisations.  The  second  and  third 
were  more  difficult.  The  fourth  was  solved  by  reducing  the  number  of 
factors  shown  at  the  and  of  the  first  chapter  using  criteria  discussed 
later  in  this  section. 

One  problem  with  asking  respondents  to  rate  a specific  system  is 
that  a number  of  respondents  may  have  no  experience  with  a specific  MIS. 
To  overcome  this,  the  survey  allows  respondents  with  no  experience  to 
rate  the  factors  for  systems  In  general,  and  those  with  experience  to 

* specific  system.  This  design  permits  testing  the  hypothesis  that 
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1C  nak«s  no  difference  which  way  Che  facCors  are  measured.  This  may  be 
useful  for  evaluacing  Che  CSU  study  and  in  guiding  fucure  efforts. 


The  Chird  problem,  obtaining  a measure  of  the  appropriate  level  for 
each  factor,  is  aore  difficult  to  address.  The  main  problem  is  that 
there  are  too  many  potentially  useful  dimensions  (factors)  relating  to 
project  success.  One  factor  used  in  this  study,  and  essentially  the  same 
as  one  used  in  the  CSU  study,  was  "identification  of  the  information 
needs  of  users."  There  are  several  useful  things  to  know  about  this 
factor.  First,  how  "well"  should  it  be  accomplished?  How  much  does  the 
factor  contribute  to  project  success  compared  to  other  factors?  Is  the 
factor  critical  in  the  sense  that,  if  it  is  not  attained  at  a minimum 
level,  the  project  will  fall?  How  "important"  is  the  factor  to  project 
success? 

Consideration  was  given  to  measuring  each  of  the  dimensions  listed. 
One  of  the  early  test  versions  of  the  survey  attempted  to  have  the 
respondent  first  indicate  if  the  factor  was  present  in  the  project  at 
about  the  level  indicated  (by  the  wording  of  the  factor).  The  respondent 
was  then  to  indicate  how  important  the  presence  or  absence  of  the  factor 
was  to  the  success  of  the  project.  A pretest  of  that  design  yielded  that 
it  was  extremely  time  consuming  for  respondents  to  complete,  and  that  the 
use  of  adjectives  to  describe  the  level  of  fulfillment  both  offended  the 
respondents  and  yielded  vague  results. 

The  design  settled  upon  through  six  different  test  version!  of  the 
survey  is  shown  in  Figure  1.  The  dimension  to  be  measured  by  this  design 
is  "how  important  each  factor  should  be  treated  in  order  for  a project  to 
be  successful."  Respondents  who  have  no  experience  with  specific  HIS 
development  efforts  rate  only  the  right  colusn,  based  upon  what  they 
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Importance  Ideal 

Given  an  Import- 

This  Project  ance 

. 56.  Control  and  coordination  of  changes. 

__ _ _ _____  37.  Identification  of  the  infomation  needs 

of  users. 

_____  58.  Canal tment  and  support  by  users. 

RATING  SCALE: 

A B C D E 

Critically  Very  Important  Somewhat  Not  Important  or 

Important  Important  Important  Not  Applicable 

Fig.  1.  Design  of  Survey  Part  C 

think  is  most  important  for  the  successful  development  of  MIS  in  general. 
Respondents  with  experience  rate  both  how  importantly  each  factor  was 
treated  during  the  project  in  the  left  colvmm,  and  how  important  the 
factor  should  have  been  considered  to  make  that  project  more  successful 
in  the  right  column.  I considered  this  dimension  to  be  probably  the  most 
beneficial  to  project  managers,  since  it  attempts  to  answer  "how  much 
attention  should  I give  to  factor  X"  and  "how  much  more  important  is 
factor  X than  factor  Y?"  This  measurement  form,  if  successful,  should 
have  the  end  result  of  allowing  project  managers  to  concentrate  their 
efforts  on  the  factors  most  requiring  their  attention. 

Hie  problem  of  including  enough  factors  was  addressed  in  Chapter  I. 
The  list  presented  at  the  end  of  that  chapter  was  viewed  with  two  objec- 
tives when  constructing  the  survey:  (1)  choose  only  the  apparently  most 
important  factors  (since  the  average  time  to  complete  the  survey  should 
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not  exceed  30  minutes  or  response  rate  will  be  low),  and  (2)  avoid 
choosing  factors  that  appear  to  be  some  expert's  technique  or  a contro- 
versial issue.  The  final  version  of  the  survey  measured  a total  of  40 
factors.  It  was  found  in  pretests  that  40  were  as  many  as  respondents 
could  rate  (in  addition  to  the  preliminary  questions  in  the  first  two 
parts)  in  the  allotted  average  30  minutes.  With  respect  to  techniques 
and  controversial  issues,  the  use  of  top-down  structured  programming, 
standardized  high-order  languages  and  a number  of  other  potential  factors 
were  not  surveyed.  Since  there  is  currently  some  controversy  over  the 
use  of  these  techniques,  it  was  felt  that  a survey  would  reveal  primarily 
that  opinion  is  polarized.  Modularization  and  integration  were  Included 
since  these  approaches  have  been  espoused  for  some  time  in  the  literature. 

Certain  of  the  factors  are  common  (with  some  rewording)  to  both  the 
CSU  study  and  this  survey.  Ihese  are  shown  in  Table  IV.  Part  of  the 
rewording  consists  of  changing  "management"  in  the  CSU  factors  rated 
second  and  eighth  to  "user"  on  this  survey.  It  was  felt  that  the  CSU 
study  implicitly  equated  the  two  terms  in  those  two  factors.  That  is, 
in  many  civilian  Institutions,  the  most  relevant  managers  are  users. 

This  may  not  be  the  case  with  all  Air  Force  development  agencies.  The 
use  of  common  factors  should  allow  evaluating  the  hypothesis  that  the 
survey  populations  are  the  same  (discussed  in  the  section  on  methods  of 
analysis). 

One  of  the  study  objectives  stated  in  the  first  chapter  was  "to 
determine  if  and  how  the  Important  factors  vary  with  a number  of  system 
and  project  attributes."  The  attributes  measured  in  Part  B of  the 
survey  were: 

System  success 
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Table  IV 

Factors  Common  to  the  CSU  Study 


Ranking  an 

CSU  Study  Wording  on  CSU  Study* 


i 


1 

2 

3 

4 

5 

6 

7 

8 
9 

10 

11 


Definition  of  objectives  of  a specific  info.  sys. 
Identification  of  info,  needs  of  agt. 

Attitude  of  mgt.  toward  a given  sys.  effort. 
Coomunication  abilities  of  the  design  team. 

Expertise  and  creative  abilities  of  the  design  team. 
Determination  of  company  objectives. 

Justification  of  system  cost. 

Participation  of  mgt.  on  the  design  team. 

Adequacy  of  time  frame  for  system  effort. 

Attitude  of  design  team  toward  user  dept. 

Attitude  of  user  dept  toward  design  team. 


Equivalent 

to  CSU  Factor  Wording  Used  on  Survey** 


1 Definition  of  the  objectives  of  the  system  (38). 

2 Identification  of  the  Information  needs  of  users  (57). 

3 Involvement  and  support  of  top  management  (33). 

4 Personal  communication  ability  of  the  designers  (48). 

5 Expertise  and  creative  ability  of  the  designers  (21). 

6 Determination  of  the  objectives  of  the  user  organization 
(25). 

7 Justification  of  system  cost  (55). 

8 Involvement  in  system  design  by  users  (47). 

9 Scheduled  time  for  the  project,  excluding  test  (40, 
test  time  measured  by  42). 

10  A positive  attitude  by  the  designers  toward  the  users 
(28). 

11  A positive  attitude  by  users  toward  designers  (49). 


* (Ref  17:10) 

**  Number  in  parentheses  after  each  factor  is  the  question  number 
as  used  on  the  survey. 


Project  size 

Single  versus  multiple  Installation 
Software-only  projects,  versus  software  and  hardware 
Number  of  user  organizations,  and  similarity  of  their 
requirements 

Application  type  (accounting,  command  and  control,  project 
management,  research) 

Technical  difficulty 

Contractor  participation 

Use  of  long-term  warranties  (on  contracts) 

Maintenance  arrangements 
Time  since  project  completed 
Whether  the  system  is  still  In  use. 

The  "system  success"  variable  was  included  to  provide  a criterion  vari- 
able for  use  In  regression  analysis  In  developing  a predictive  model. 

Size  was  Included  since  many  experts  have  argued  that  there  are  problems 
Involved  In  large  projects  that  are  not  Important  In  smaller  ones.  The 
number  of  installations  and  user  organizations,  as  well  as  the  similarity 
of  user  requirements  was  measured  because  these  factors  would  appear  to 
Impact  both  on  the  ease  with  which  users  may  be  involved  In  the  project, 
and  the  need  for  Involvement.  Other  hypotheses  to  be  tested  Include  that 
the  factor  ratings  vary  slgnlf lcantly  for  software-only  projects,  with 
application  type,  with  the  degree  of  contractor  participation,  and  with 
the  degree  of  technical  difficulty. 

Maintenance  arrangements,  use  of  long-term  warranties  and  project 
organization  were  Included  primarily  as  candidate  regression  (Independent) 
variables.  A number  of  experts  have  cited  both  project  organization  and 
maintenance  arrangements  as  Important  factors.  However,  since  It  Is  not 
well  agreed  what  form  of  organization  or  maintenance  arrangement  la  best, 
these  factors  are  best  measured  as  nominal  variables  rather  than  con- 
tinuous ordinal  factors.  This  is  why  the  two  were  Included  In  Part  B 
of  the  survey.  The  use  of  long-term  maintenance  warranties  was  Included 


I 


28 


sine*  that  procedure  vas  strongly  recommended  by  a study  performed  for 
the  Department  of  Defense  in  1974  (Ref  9:17). 

Description  of  the  Population 

Software  development  and  the  development  of  MIS  in  the  Air  Force 
are  performed  by  a wide  variety  of  organisations  for  many  different  pur- 
poses. Figure  2 is  intended  to  illustrate  this  variability  by  showing 
some  of  the  Air  Force  agencies  involved  in  MIS  development.  Individual 
bases,  on  the  one  extreme,  develop  systems  for  local  users.  At  the  other 
extreme,  the  Air  Force  Data  Systems  Design  Center  (AFDSDC) , under  the 
Air  Force  Data  Automation  Agency  (AFDAA)  is  responsible  for  developing 
and  maintaining  standard  automated  data  systems  for  use  throughout  the 
Air  Force.  This  currently  involves  196  systems  operating  an  350  com- 
puters worldwide  (Ref  38:1).  The  Air  Force  Military  Personnel  Center 
(AFPMC)  is  responsible  for  all  common  use  personnel  systems.  The  Elec- 
tronic Systems  Division  (ESD)  of  Air  Force  Systems  Corns land  (AFSC)  becomes 
involved  in  many  larger  projects.  Each  major  command  (MAJCOM)  develops 
systems  for  its  own  unique  requirements. 

These  systems  also  vary  considerably  both  in  size  and  use.  For 
example,  in  a survey  of  25  System  Development  Corporation  and  Department 
of  Defense  project  managers,  "Project  size  varied  from  2 persons  and 
5000  object  instructions  to  200  persons  and  1.2  million  instructions  . . . 
Projects  ranged  from  8 months  to  6 years  in  length"  (Ref  35:2). 
Additionally,  systems  in  the  Air  Force  vary  from  relatively  familiar 
accounting  and  inventory  applications  to  extremely  sophisticated  command 
and  control  applications. 
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Fig.  2.  Some  Air  Force  Organizations  Involved  In  MIS  Developaent 
(Not  an  official  listing  or  organizational  chart) 

Thus  one  problem  In  determining  the  survey  sample  was  to  obtain  data 
from  the  full  range  of  development  types  and  system  uses.  Additionally, 

It  would  have  been  ideal  to  obtain  responses  from  users  as  well  as 
developers  of  MIS.  Unfortunately  it  Is  probably  Impossible  to  Identify 
an  adequate  sample  of  "experienced  users."  Therefore  the  approach  taken 
was  to  select  developaent  agencies  to  be  surveyed.  The  agencies  selected 
were  the  AFOSOC  at  Gunter  AFS,  Alabama,  Air  Force  Logistics  Command  (AFLC) 
activities  at  Wright-Patterson  AFB,  Ohio  and  Air  Force  Systems  Command 
activities  (Aeronautical  Systems  Division,  Foreign  Technology  Division 
and  others  Including  the  Air  Force  Avionics  Laboratory,  Air  Force 
Materials  Laboratory  and  Aero-Propulsion  Laboratory)  at  Wrlght-Patterson 
AFB,  Ohio.  Also  surveyed  was  Data  Automation  at  Headquarters,  Strategic 
Air  Comand  (SAC),  Offutt  AFB,  Nebraska. 

Although  this  sample  is  biased  toward  agencies  with  a large  Invest- 
sent  In  software  development,  responses  indicated  a fair  percentage  of 
smaller  efforts,  and  several  of  the  units  surveyed  at  Wrlght-Patterson 
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are  involved  in  base-level  type  activities.  These  units  also  reflect  a 
complete  cross  section  in  terns  of  systeas  application.  SAC  is  heavily 
involved  in  command  and  control,  AFLC  in  accounting- type  applications, 
and  APSC  in  project  aanagement  and  research.  AFOSOC  was  Included  to 
find  the,  possibly  unique,  problems  of  a developer  of  standardised 
systems. 

The  next  problem  was  to  identify  the  personnel  to  be  surveyed.  It 
was  decided  at  this  point  to  survey  only  personnel  of  a rank  high  enough 
so  that  they  would  probably  be  aware  of  managerial  problems  as  well  as 
technical  issues.  For  that  reason  the  sample  was  limited  to  Air  Force 
officer  personnel  and  civil  servants  in  grades  GS-9  and  above.  It  may 
have  been,  in  retrospect,  desirable  to  include  senior  noncommissioned 
officers  (grades  E-8  and  E-9).  The  AFDSDC,  in  particular,  has  a number 
of  highly  qualified  personnel  in  that  category,  some  in  policy  making 
positions. 

Because  there  is  no  single  pool  of  personnel  working  in  systems  de- 
velopment at  Wright-Patterson  APB,  personnel  selected  to  participate  at 
that  base  were  limited  to  "computer  specialists."  These  were  Air  Force 
specialty  codes  51XX  and  civil  service  codes  33X  and  C1320  (C1320  is 
mathematician,  but  most  at  Wright-Patterson  work  primarily  with  computers). 
The  sample  was  not  limited  by  specialty  code  at  AFOSOC  and  SAC,  since 
personnel  working  with  systems  development  at  those  locations  are  pri- 
marlly  all  in  one  organisation,  and  therefore  easily  identified.  Also, 
since  both  of  those  organizations  have  chosen  to  facilitate  user  involve- 
ment by  including  functional  specialists,  this  gave  the  opportunity  to 
gain  more  of  a "user's  perspective"  in  the  data.  The  AFDSDC,  in 
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particular,  makes  a practice  of  obtaining  highly  qualified  functional 
(for  example  accounting,  logistics,  operations,  medical)  personnel.  These 
personnel  often  serve  a tour  of  duty  at  AFDSDC  working  in  system  develop- 
ment, and  subsequently  spend  much  of  their  careers  alternating  between 
their  functional  specialties  in  the  field  and  AFDSDC.  SAC  also  has  a 
number  of  personnel  who  are  highly  qualified  in  both  a functional  specialty 
and  systems  development. 

Approximately  40%  or  750  of  the  nearly  1900  personnel  working  in 
computer-related  duties  (in  the  appropriate  grades)  at  the  three  loca- 
tions were  chosen  to  participate  in  the  survey.  Selection  was  made 
randomly.  The  sample  size  was  large  because  of  the  need  to  maintain  an 
adequate  cell  count  throughout  the  analysis.  If,  for  example,  the  opinions 
of  inexperienced  personnel  (in  terms  of  either  MIS  projects  or  years 
working  with  computers)  are  significantly  different  when  other  variables 
are  controlled,  it  may  be  necessary  to  segregate  those  responses  from  the 
rest  of  the  data.  The  seven  demographic  questions  of  Part  A of  the  survey 
provide  a capability  to  segregate  the  data  based  upon  command,  years  ex- 
perience, specialty,  job  orientation,  grade,  user  contact,  and  MIS  pro- 
ject experience,  or  any  combination  of  these  variables. 

One  problem  in  designing  Part  3 of  the  survey  originated  from  the 
inclusion  of  relatively  lower-grade,  technically-oriented  personnel  in 
the  sample.  It  would  have  been  ideal,  for  example,  to  obtain  relatively 
precise  measures  of  system  size  and  cost.  It  was  found  during  the  pre- 
test however  that  the  technically-oriented  respondent  had  little  idea  of 
system  cost,  and  often  a not  much  better  idea  of  system  size  in  terms  of 
precise  measures  like  lines  of  code,  or  man-years,  A related  problem 
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was  found  when  using  relatively  precise  measures  of  size  with  respondents 
whose  experience  involved  only  small  projects.  Size  is  not  logically 
measured  on  a linear  scale.  Recalling  the  project  manager  survey  cited 
earlier,  the  largest  project  included  in  that  survey  involved  200  people, 
for  example  (Ref  35:2).  A linear  scale  with  five  possible  responses  and 
a maximum  value  of  "200  or  more"  would  have  responses  at  40,  80,  120  and 
160.  Well  over  90%  of  Air  Force  projects  would  fall  into  the  first  cate* 
gory,  but  a few  extremely  Important  projects  would  be  included  in  the  last. 
Thus  a nonlinear  scale  with  intervals  at  5,  10,  20,  50,  100  and  200  (for  < 
contractor  personnel)  was  attempted  on  the  first  pretest.  The  use  of 
such  a scale  made  the  survey  seem  very  imposing  to  those  involved  in 
only  very  small  projects  and  they  were  then  unmotivated  to  complete  the 
survey.  That  experience  also  taught  that  technically-oriented  personnel 
often  have  very  little  idea  how  many  personnel  a contractor  had  working 
on  a project.  The  final  version  of  the  survey  utilized  subjective 
judgment  of  size,  based  upon  the  number  of  people  and  the  duration  of 
the  project.  These  were  some  of  the  considerations  which  led  to  the 
final  design  of  survey  Part  B. 

The  surveys  were  administered  during  September,  1977.  The  surveys 
were  hand  carried  to  most  of  the  organizations  at  Wright-Patterson  AFB, 
and  1 gained  the  support  and  cooperation  of  the  appropriate  supervisor 
at  each  organization  visited.  Although  participation  was  voluntary,  past 
survey  efforts  have  found  that  response  rates  are  higher  when  supervisors 
indicate  that  they  support  the  project.  The  surveys  administered  at  Hq. 
SAC  were  mailed  to  a project  officer  who  made  distribution.  Respondents 
returned  the  surveys  directly  through  the  mall  upon  completion.  The 
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researcher  was  able  to  personally  visit  the  AFDSDC.  That  organisation 
had  appointed  a central  project  officer  to  assist  in  making  necessary 
arrangements,  and  each  Directorate  had  In  turn  appointed  a project 
officer  to  distribute  and  explain  the  survey  to  the  respondents.  This 
left  the  researcher  relatively  free  and  the  opportunity  was  taken  to 
visit  each  of  the  Directorates  and  conduct  personal  Interviews. 

Nearly  all  of  the  AFDSDC  Directorate  chiefs  or  their  deputies  (in 
some  cases  both)  were  interviewed  during  a three-day  period.  A list  of 
personnel  interviewed  is  contained  in  Appendix  B.  The  interviews  were 
performed  with  the  understanding  that  the  Interviewees  would  not  be 
quoted  directly,  specific  systems  would  not  be  mentioned  by  name  and 
that  only  summary  information  would  be  presented  in  this  study.  The 
questions  asked  are  shown  in  Figure  3. 

There  were  three  objectives  in  performing  the  interviews:  to  obtain 
the  views  of  top-level  managers  concerning  what  factors  are  most  Impor- 
tant for  developing  Air  Force  MIS,  to  obtain  information  about  what 
factors  emerge  as  most  important  outside  the  confines  of  a highly 
structured  survey,  and  to  obtain  a better  idea  if  and  in  what  ways  MIS 
development  in  the  AFDSDC  and  the  Air  Force  may  vary  from  MIS  development 
in  the  civilian  community.  The  results  of  the  interviews  are  contained 
in  the  next  chapter  along  with  the  results  of  the  survey. 

Methods  of  Analysis 

The  analysis  plan  followed  can  be  divided  into  four  parts:  initial 
investigation  of  the  data,  analysis  of  the  factor  ratings,  factor 
analysis  (in  the  statistical  sense)  and  regression  analysis.  During  the 
initial  investigation  of  the  data  a simple  cross-tabulation  technique  was 
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1.  How  would  you  compare  computerized  management  information  systems 
In  the  Air  Force  to  those  in  civilian  organizations? 

a.  In  terms  of  the  nature  of  the  systems? 

b.  In  terms  of  the  development  process? 

2.  Do  you  think  that  the  development  of  Air  Force  systems  requires  the 
use  of  procedures,  management  emphasis  or  skills  which  are  dif- 
ferent from  those  required  to  develop  civilian  systems?  If  so. 

In  what  ways? 

3.  Has  your  Directorate  ever  been  involved  in  the  development  of 
management  information  systems?  If  so,  were  there  any  problems 
which  you  think  were  peculiar  to  your  particular  mission  or 
organizational  structure? 

A.  What  "factors"  are  most  Important  to  the  successful  development  of 
Air  Force  management  information  systems?  Are  these  factors 
different  for  AFDSDC? 

5.  Recall  the  most  successful  management  information  system  develop- 
ment effort  with  which  you  are  familiar.  Briefly  describe  the 
project  in  terms  of  size.  Intended  use,  technical  difficulty, 
organizational  structure  and  contractual  arrangements. 

a.  What  "factors"  were  moat  Important  to  the  success  of  that 
system?  Please  rank-order  the  factors  (1st,  2nd,  ...)  in 
terms  of  importance. 

6.  Recall  the  least  successful  management  information  system  develop- 
ment effort  with  which  you  are  familiar.  Briefly  describe  the 
project  in  terms  of  size.  Intended  use,  technical  difficulty, 
organizational  structure  and  contractual  arrangements. 

a.  What  "factors"  were  most  important  to  the  failure,  or  lack  of 
success  of  that  system?  Please  rank-order  the  factors  (1st, 
2nd,  ...)  in  terms  of  importance. 


Fig.  3.  Format  Used  in  Interviewing  AFDSDC  Personnel 


used  to  determine  the  cell  counts  for  various  combinations  of  demographic 
variables  and  system  attributes  (Parts  A and  B of  the  survey).  A typi- 
cal output  of  this  analysis  would  be  that  there  were  n "very  large," 
"technically  difficult,"  "successful"  systems  Included  in  the  survey 
responses  (n  respondents  Indicated  that  particular  combination  in  Part  B). 
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This  also  gave  some  initial  feel  for  any  trends  (correlation  or  biasing) 
in  the  data. 


Cross-tabulation  was  followed  by  the  use  of  the  Automatic  Inter- 
action Detection  (AID)  algorithm  with  System  Success  as  the  criterion 
variable.  The  AID  algorithm  operates  by  dividing  the  total  sample  into 
the  most  statistically  homogeneous  groupings  possible  relative  to  the 
criterion  variable.  The  measure  of  homogeneity  is  the  sum  of  the  squared 
differences  between  the  response  for  each  individual  in  the  group  and  the 
group  mean  (called  the  sum  of  squares).  For  example,  if  Size  were  used 
as  the  independent  variable  for  a split,  the  algorithm  might  split  the 
total  sample  into  two  groups:  Group  2 consisting  of  responses  A through 
0 ('Very  small"  to  "fairly  large");  and  Group  3 consisting  of  responses 
C and  F ("large"  and  'Very  large"),  the  mean  value  of  Success  is  calcu- 
lated for  each  group,  and  then  the  sum  of  squares  for  each  group  is  cal- 
culated (Ref  40:24-26). 

The  algorithm  operates  in  a two-stage  process  to  determine  the  "best" 
split  of  a group.  First  it  determines  the  best  split  for  each  independent 
variable  by  calculating  the  sum  of  squares  for  all  possible  splits, 
where  a "possible"  split  is  one  which  results  in  two  mutually  exclusive 
groups  that  are  exhaustive  of  all  available  responses.  The  sum  of  squares 
is  then  calculated  and  the  split  resulting  in  Che  smallest  sum  of  squares 
is  selected  as  best.  The  algorithm  then  repeats  the  procedure  for  the 
remainder  of  the  independent  variables  and  the  variable  with  the  best 
split  of  the  entire  set  is  selected  to  split  the  group.  The  procedure 
is  repeated  until  preset  limits,  such  as  group  size,  are  reached.  For 
example,  if  the  first  split  were  made  on  Size  as  described  above,  Group  2 
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night  subsequently  split  on  Preliminary  Design  into  Groups  4 and  5,  and 
Group  3 might  split  on  Education  and  Experience  of  the  Project  Manager 
into  Groups  6 and  7.  The  results  of  the  process  nay  be  displayed.  If 
desired,  as  a tree  diagram  (Ref  40:24-26). 

Che  advantage  of  AID  in  this  particular  application  is  that  It 
allows  the  use  of  nominal  variables  (such  as  a number  of  the  questions 
on  Part  A and  B of  the  survey)  without  requiring  the  analyst  to  identify 
and  compute  dummy  variables.  One  potential  drawback  is  that  AID  may  re- 
sult in  idiosyncratic  results  in  smaller  samples  such  as  the  one  used  in 
this  survey,  as  found  by  Gooch  (Ref  41:71).  The  Intent  of  using  AID  in 
this  particular  application  is  both  to  provide  an  initial  indication  of 
any  unproductive  paths  planned  a priori  in  the  remainder  of  the  modeling 
effort  and  to  discover  trends  which  may  not  be  apparent  from  a cursory 
examination  of  the  data. 

Three  different  techniques  were  used  in  the  investigation  of  the 
factor  ratings.  These  were  t-tests,  analysis  of  variance  (ANOVA)  and 
the  Spearman  Rank-Order  Correlation  coefficient.  Given  the  aggregate 
factor  ratings  (Part  C of  the  survey)  for  the  entire  sample,  the  mean 
and  variance  for  each  factor  was  computed.  The  sample  was  then  broken 
down  into  groups  to  test  the  hypothesis  that  the  mean  ratings  for  each 
group  for  each  factor  were  equal.  For  those  Independent  variables  used 
to  divide  the  sample  into  groups  which  could  logically  be  divided  Into 
only  two  groups,  the  t-test  was  used.  This  was  used,  for  example  , to 
determine  if  those  respondents  who  rated  the  factors  for  a specific 
system  differed  significantly  from  those  who  rated  the  factors  only  in 
general. 
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IC  was  necessary  Co  use  ANOVA  for  Chose  independenC  variables  chac 
logically  had  co  be  divided  into  more  chan  two  groups,  such  as  command 
of  assignmenc  of  Che  respondent  The  ANOVA  procedure  is  based  upon 
decomposing  Che  variance  in  Che  dependent  variable  (the  factor  ratings 
in  Fart  C of  the  survey)  into  the  portions  due  to  Che  presence  of  the 
independent  variable,  say  command,  and  that  due  to  random  effects  or 
error.  The  F-statistic  is  Chen  used  to  test  the  hypothesis  chat  Che 
means  are  equal.  This  procedure  was  used  carefully,  however  since  the 
F-statistic  will  permit  the  hypothesis  chat  there  is  no  difference  to  be 
rejected  even  if  only  one  category  is  significantly  different.  For 
example,  if  information  needs  were  Che  dependent  variable,  and  each  of 
the  commands  placed  about  Che  same  mean  importance  on  Information  needs 
with  the  exception,  of  AFLC,  the  F-statistic  causes  the  hypothesis  to  be 
rejected.  If  this  hypothetical  situation  were  found  for  a number  of  the 
factors  when  evaluated  by  command,  it  would  be  an  Indication  that  fur- 
ther analysis  was  needed  to  determine  why  AFLC  was  different. 

It  vas  necessary  to  use  Che  Spearman  rank-order  correlation  co- 
efficient , a nonparamecrlc  statistic,  to  test  the  hypothesis  Chat  the 
Air  Force  population  included  in  this  survey  is  Che  same  as  Che  non-Air 
Force  population  surveyed  by  Che  CSU  researchers.  This  procedure  Is  used 
by  treating  Che  two  sets  of  results  (CSU  and  this  study)  as  variables 
and  the  ranks  of  the  11  common  factors  (Table  IV)  as  cases.  For  example, 
of  the  common  factors,  information  needs  might  be  ranked  1,  and  objec- 
tives 2.  The  data  is  paired  rank  against  rank  for  both  studies  (for 
example  (1,3), (2, 4), (3,1)  for  all  11  faccors),  and  the  correlation  co- 
efficient Is  a measure  of  how  well  the  rank  on  the  one  study  predicts  Che 
rank  on  the  other  study.  This  procedure  also  produces  a statistic  which 
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is  tasted  for  significance  against  the  ^distribution  with  n-2  (9) 
degrees  of  freedom.  In  the  event  that  little  correlation  exists  between 
the  two  studies,  the  analysis  can  also  be  attempted  without  factors  3,  8, 
and  9 since  the  difference  in  wording  between  the  two  surveys  is  greatest 
for  those  factors.  It  should  be  added  that  this  procedure  cannot  prove 
conclusively  that  the  two  populations  are  truly  different,  only  that  the 
populations  are  the  same  (as  far  as  how  the  factors  were  ranked)  or  that 

I 

the  factor  ranks  are  not  the  same.  This  is  due  both  to  the  nature  of  the 
hypothesis  and  because  the  survey  designs  are  completely  different. 

The  third  and  fourth  steps  in  the  analysis  were  the  use  of  factor 
analysis  and  multiple  regression.  Factor  analysis  was  used  both  as  a 
data  reduction  technique  and  to  search  for  underlying  relationships  in 
the  data.  For  example,  the  CSU  researchers  used  factor  analysis  to  re- 
duce  the  14  variables  on  their  factor  checklist  to  four  "critical" 
factors  which  they  call  planning  and  control,  attitude,  expertise  and 
involvement  (Ref  17:14).  The  procedure  operates  by  finding  associations 
between  the  variables.  Each  set  of  associations  is  developed  such  that 
it  is  statistically  Independent  from  every  other  set  of  associations 
built  by  the  procedure.  These  associations  are  used  to  define  the 
resultant  factors.  Recalling  Table  III  in  Chapter  I,  "Factors  Relating 
to  User  Involvement,"  it  is  relatively  apparent  that  a number  of  poten- 
tially Important  factors  relate  to  this  potential  "critical”  factor. 

This  researcher  felt  a priori  that  factors  31,  33,  37,  47,  and  38  re- 
lated to  user  Involvement.  Other  potential  critical  factors  hypothesised 
a priori  related  to  the  "front  end"  of  the  project  (requirements  defini- 
tion, initial  analysis,  determination  of  user  objectives,  system  defini- 
tion), control,  attitude,  planning  and  expertise.  The  underlying 
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rationale  for  using  factor  analysis  is  that,  given  the  underlying 
relationships  between  the  variables  which  define  the  factor*  the  really 
critical  thing  for  the  project  manager  to  concentrate  Is  the  level  of 
the  factor.  The  results  of  the  factor  analysis  were  used  in  multiple 
regression*  along  with  a number  of  other  potential  Independent  variables* 
in  the  effort  to  develop  specific  models  for  predicting  system  success. 
Further  discussion  of  the  modeling  effort  is  reserved  for  the  next 
chapter*  along  with  the  results  of  the  analysis. 
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III.  Results 


Interview  Results 

As  discussed  In  the  last  chapter,  a series  of  personal  interviews 
were  performed  while  administering  surveys  at  the  AFDSDC,  Gunter  AFS, 
Alabama,  using  the  format  shown  In  Fig.  3 (Page  33).  The  objectives  In 

I 

performing  the  interviews  were:  to  obtain  the  views  of  top  level 
managers  concerning  what  factors  are  most  important  for  developing  Air 
Force  MIS;  to  obtain  Information  about  what  factors  emerge  as  most  Im- 
portant outside  the  confines  of  a structured  survey;  and  to  obtain  a 
better  understanding  of  how  MIS  development  in  the  AF3SDC  and  the  Air 
Force  differs  from  MIS  development  in  the  civilian  community. 

The  views  of  the  interviewees  with  respect  to  the  most  Important 
factors  are  summarized  in  Table  V.  The  classification  of  these  factors 
into  the  areas  of  user  Involvement,  requirements  definition,  development 
approach,  resources,  organization  or  technical  Issues  is  based  upon  the 
perceived  intent  of  the  Interviewees.  The  factors  listed  are  those 
mentioned  by  two  or  more  interviewees.  The  numbers  listed  in  the  first 
column  to  the  right:  of  each  factor  indicate  how  many  Interviewees  cited 
the  factor  as  being  important  to  the  successful  development  of  systems 
in  general.  The  second  and  third  columns  indicate  respectively,  the 
number  of  persons  listing  the  factors  as  of  primary  Importance  for  the 
success  of  specific  successful  systems  and  the  lack  of  success  of 
unsuccessful  systems.  The  results  of  the  interviews  were  surprising  to 
this  researcher  in  terms  of  the  amount  of  emphasis  placed  upon  require- 
ments definition  activities  and  upon  the  use  of  a structured  development 
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Table  V 

Summary  of  Interview  Results 


Number  of  Interviewees  Citing  as 
Important : 


To  To 

In  Successful  Unsuccessful 

Factor  General  Systems  Systems 


throughout  the  project  4 4 

during  requirements  definition  3 

by  Including  functional  users  In 


development  team  4 2 

in  planning  2 

determining  the  information  needs 

of  user  management  1 2 


clear,  formal 

understanding  of  requirements 


formalized,  with  reviews 

5 

2 

reviews  at  all  levels,  involve  user 

1 

2 

formalized  change  control 

3 

3 

incremental  development 

2 

2 

control,  visibility 

1 

2 

Resources 

1 

3 

ability,  attitude  of  the  developers 
ability  of  Che  project  manager 

6 

4 

3 

top  management  support 

2 

3 

1 
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geographically  separated  users.  The  use  of  a structured  development 
approach  has  been  receiving  increased  attention  in  the  literature  in 
recent  years.  The  AFDSDC  has  developed  and  applied  one  such  approach 
through  the  medium  of  AFDSDC  Manual  300-3.  Although  some  of  the  per- 
sonnel contacted  considered  the  manual  as  somewhat  controversial,  a 
number  of  the  directorate  chiefs  and  their  deputies  Indicated  that  the 
concepts  and  procedures  involved  were  critical  to  systems  success.  The 
manual  was  a point  of  pride  among  some  of  the  interviewees  who  considered 
it  revolutionary  in  terms  of  disciplining  the  development  process. 

Another  point  of  Interest  concerning  the  Interview  results  was  the 
relative  distribution  of  the  factors  between  those  important  to  success 
and  those  causal  of  failure.  Although  the  classification  scheme  is  some- 
what arbitrary,  it  is  Interesting  to  note  that  user  Involvement  and 
resources  were  associated  primarily  with  successful  efforts  and  the 
(lack  of)  requirements  definition  and  a disciplined  development  approach 
with  unsuccessful  efforts.  Additionally,  the  "technical  Issues"  arose 
only  in  conjunction  with  less  successful  systems.  This  finding  provides 
sane  indication  that  It  may  be  useful  in  future  efforts  to  take  a two 
model  (success  and  failure)  approach  to  modeling  MIS  efforts.  This  is 

discussed  more  fully  in  the  section  on  the  "Predictive  Modeling  Efforts." 

- 

The  interviewees  in  general  felt  that  there  were  few  significant 

i 

differences  betveen  system  development  in  the  Air  Force  or  at  the  AFDSDC 
and  in  the  civilian  community.  The  most  frequently  cited  sources  of 
difference  betveen  the  Air  Force  and  the  civilian  community  were 

(followed  by  the  number  of  personnel  citing  the  reason): 

1.  Civilian  project  managers  are  less  constrained,  given  greater 
flexibility.  (4) 
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The  Air  Force  must  make  resource  decisions  further  In  advance, 
making  long  range  planning  more  important.  (3) 

3.  The  Air  Force  project  manager  has  less  control  over  personnel 
resources.  In  tens  of  both  availability  and  quality.  (2) 

1 

4.  The  Air  Force  is  less  concerned  with  cost  than  with  mission 
effectiveness,  as  opposed  to  business  organisations  that 
attempt  to  quantify  benefits  in  tens  of  more  measurable 
profits.  (2) 

Similarly,  the  lntarvlevees  felt  that  the  AFDSDC  was  not  significantly 
different  from  the  rest  of  the  Air  Force.  No  single  reason  was  cited  by 
more  than  one  person. 

Summary  Data  from  the  Survey 

A total  of  456  of  the  original  730  surveys  vere  returned  in  a fon 
suitable  for  Inclusion  in  the  data.  This  represents  a 60.8%  return  rate 
with  little  followup  and  no  second  mailing.  Of  the  total,  176,  146,  87, 
and  &'«  respectively  were  received  from  AFDSDC,  AFLC,  SAC  and  AFSC. 

Command  could  not  be  detenined  on  three  fons.  The  average  respondent 
had  at  least  10  years  experience  in  the  computer  systems  career  field, 
and  123  respondents  Indicated  that  they  had  more  than  13  years  experi- 
ence. The  average  civilian  grade  was  GS-12  and  the  average  military 
respondent  was  somewhat  above  0-3 . A total  of  280  respondents  were 
civilian  and  166  military.  A total  of  79  respondents  formally  belonged 
to  a specialty  not  designated  as  a full-time  computer  specialty.  This 
gives  some  idea  of  the  degree  of  "user"  participation  in  the  survey. 

The  respondents  were  about  evenly  split  between  managers  and  technicians, 
although  only  126  classified  their  jobs  as  involving  primarily  new 
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(as  opposed  to  existing)  systems.  The  average  respondent  felt  that 
their  jobs  brought  them  Into  contact  with  users  at  least  "often”  and  had 
been  involved  in  at  least  two  MIS  development  efforts  In  the  last  ten 
years.  A total  of  126  indicated  no  experience  with  MIS  development  and 
therefore  did  not  complete  Part  B,  and  completed  Part  C only  for  systems 
in  general.  The  distribution  of  the  responses  to  Part  A and  Part  B 
questions  is  shown  in  Appendix  C. 

As  discussed  in  the  last  chapter,  the  design  of  Part  C of  the  survey 
is  such  that  respondents  who  have  not  been  involved  in  one  or  more  MIS 
development  efforts  rate  each  of  the  factors  in  terms  of  how  important 
that  factor  should  Ideally  be  treated  for  the  successful  development  of 
MIS  in  general.  Respondents  who  have  been  Involved  in  specific  efforts 
rate  each  factor  in  terms  of  how  important  the  factor  Ideally  should 
have  been  treated  for  a specific  effort,  and  how  important  they  feel  it 
actually  was  treated.  The  means  and  ranks  of  the  top  30  variables  on  the 
ideal  plane  are  shown  in  Table  VI.  The  possible  range  for  each  variable 
is  from  5.0  or  "Critically  Important"  to  1.0,  "Not  Important  or  Not 
Applicable"  (tero  was  reserved  for  missing  data  not  Included  in  the 
calculations).  The  left  column  shows  the  means  and  ranks  for  the  total 
sample,  the  middle  the  ratings  by  experienced  respondents,  and  the  right 
the  ratings  by  inexperienced  respondents. 

As  is  apparent,  the  ratings  are  fairly  close  between  the  two  groups. 
Using  the  t-test  of  significance  for  differences  between  means,  only  10 
of  the  entire  40  variables  and  7 of  the  30  shown  in  Table  VI  vary 
significantly  at  the  .05  level.  These  are  (question  number  shown  in 
parentheses) : 

User  Experience  with  Computers  (26) 

Use  of  Automated  Development  Tools  (27) 


Table  VI 

and  Rank  of  Top  30  Part  C Variables,  Ideal  Ratings,  for  Total  Sasiple 
MIS  Experienced  and'  Inexperienced  Respondents 
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Table  VI  (continued) 

Mean  and  Rank  of  Top  30  Part  C Variables,  Ideal  Ratings,  for  Total  Sample 
MIS  Experienced  and  Inexperienced  Respondents 


Designer  Attitude  Toward  Users  (28) 

Use  of  a Disciplined  Developnent  Approach  (29) 

Scheduled  Time  for  the  Project  (40) 

Thorough  Preliminary  Design  (41) 

User  Involvement  in  Design  (47) 

User  Attitude  Toward  Designers  (49) 

Criteria  for  Continuing  the  Project  (50) 

Contractor  Responsiveness  (52) 

The  inexperienced  respondents  rated  26 , 27,  28,  50  and  52  higher  and 
the  remainder  lower.  Variable  52,  Contractor  Respons iveness , cannot  be 
appropriately  compared  since  only  about  half  of  the  experienced  respon- 
dents rated  systems  in  which  contractors  were  involved.  Using  the 
Spearman  Rank  Order  Correlation  as  a summary  comparison  of  the 
inexperienced  and  experienced  groups  (across  all  40  variables),  a cor- 
relation of  .8584  was  found,  which  is  significant  at  the  .001  level. 

Thus  the  hypothesis  that  it  makes  no  difference  which  way  the  entire  set 
of  variables  (as  a group)  is  measured  on  the  ideal  plane  is  acceptable. 
This  is  somewhat  more  significant  given  the  relative  compression  of  most 
of  the  variables  on  the  rating  scale.  The  minimum  rating  of  the  30 
variables  shown  (for  the  total  sample)  is  3.59,  or  more  than  half  way 
between  "important"  and  "very  important. " Only  three  (30,  26  and  27) 
of  the  variables  were  rated  less  than  3.0  for  the  total  sample,  and  four 
(same  three  with  52)  for  the  experienced  group. 

Those  variables  common  to  both  this  effort  and  the  CSU  study  are 
indicated  by  asterisks  next  to  the  factor  name  in  Table  VI.  As  can  be 
seen,  the  CSU  researchers  Independently  Identified  the  three  most 
important  variables  which  emerged  from  the  results  of  this  survey;  and 
the  top  four  which  emerged  from  the  experienced  respondents.  The 
remainder  of  the  CSU  variables  are  scattered  throughout  the  remaining 
ranked  variables  used  in  this  study. 
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The  Spearman  Rank  Order  Correlation  was  used  to  compare  the  results 
of  the  two  studies  over  the  11  common  variables,  as  discussed  In  Chapter 
II.  The  results  of  this  procedure  are  shown  in  Table  VII  for  the  total 
sample,  experienced  and  inexperienced  respondents.  As  can  be  seen,  none 
of  the  correlations  are  significant  at  the  .03  level,  and  the  correlation 
worsens  when  only  the  eight  "most  cotmnon"  are  examined.  Other  groups 
were  examined  for  closer  correlations.  One  of  these,  the  ideal  ratings 
for  systems  that  were  single  (as  opposed  to  multiple)  installations, 
correlated  significantly  at  the  .048  level,  which  makes  sense  intuitively 
in  that  most  of  the  CSU  respondents  were  probably  involved  in  single 
installation  projects.  However,  given  that  the  correlation  is  close  to 
statistical  significance  and  that  only  ll  variables  are  Involved,  the 
search  for  closer  correlations  is  a dubious  procedure.  For  example, 
while  examining  the  hypothesis  that  accounting  systems  would  correlate 
more  closely,  this  researcher  instead  found  that  the  opposite  was  true 
and  that  command  and  control  systems  correlated  at  the  .028  level.  In 
summary,  it  cannot  be  concluded  that  the  two  populations  are  the  same  or 
that  the  method  of  measurement  does  not  matter. 

However  the  correlations  are  relatively  close  to  significance,  so 
that  it  probably  can  be  concluded  that  there  are  strong  similarities. 
Examining  Table  VII,  the  primary  areas  of  difference  concern  (question 
numbers  in  parentheses)  User  Objectives  (25),  Attitudes  (28,49)  and 
Justification  of  Cost  (55).  The  Objectives  of  the  User  Organisation  were 
rated  first  of  the  Part  C variables  by  nearly  all  groups  analysed  by  this 
researcher.  This  corresponds  nicely  with  my  hypothesis  stated  in  Chapter 
I that  the  process  of  setting  objectives  is  more  difficult  in  the  Air 
Force  chan  in  civilian  organisations.  One  counter  explanation  might  be 
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Table  VII 

Comparison  with  the  CSU  Study 


Variable  Name (Nr) 

CSU  Rank 

Rank  by 
Total 

Rank  by 
Exper 

Rank  by 

In exper 

System  Objs  (38) 

1 

2 

3 

2 

Info  Needs  (57) 

2 

3 

2 

3 

*Top  Mgt  Supp  (33) 

3 

6 

6 

5 

Comm  Ability  (48) 

4 

8 

8 

1 

Design  Expert  (21) 

**5 

4 

4 

4 

User  Objs  (25) 

*★6 

1 

1 

1 

Justlf  of  Cost  (55) 

7 

11 

11 

9 

♦Users  Design  (47) 

8 

9 

9 

10 

♦Project  Time  (40) 

9 

10 

10 

11 

Designer  Att  (28) 

10 

5 

5 

6 

User  Attitude  (49) 

11 

7 

7 

8 

Spearman  correlation  for  11  vars: 

.506 

.515 

.465 

Significance: 

.057 

.053 

.075 

Spearman  corr  for  8 

most  common: 

.491 

.467 

.431 

Significance: 

.109 

.122 

.144 

♦Three  factors  excluded  in  the  analysis  of  the  8 moat  common  variables. 
♦♦Tied  ranks. 


that  the  respondents  associated  or  confused  System  Objectives  (38)  with 
User  Objectives  (23).  Were  this  the  case,  it  would  be  expected  that  the 
two  variables  would  load  on  the  same  factor  during  factor  analysis. 

However  that  was  the  case  in  only  one  of  three  primary  factor  analyses, 
as  is  discussed  in  that  section.  The  fact  that  Justification  of  Cost 
(35)  was  rated  as  less  Important  in  these  results  than  in  the  CSU  results 
corresponds  with  the  views  of  some  interviewees  discussed  earlier.  It 
may  well  be  that  the  emphasis  on  mission  effectiveness  and  the  relatively 
unquantlf iable  nature  of  the  benefits  of  that  goal  override  the  Justifl* 
cation  of  Cost  in  the  Air  Force.  One  explanation  for  the  greater  emphasis 
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placed  upon  Designer  and  User  AtClCudes  by  the  Air  Force  respondents 
alght  be  that  the  greater  physical  separation  between  developer  and  user 
organizations  in  the  Air  Force  requires  greater  use  of  informal  rather 
than  formal  authority.  That  is,  since  it  would  usual ly  be  necessary  to 

I 

appeal  to  a very  high  organizational  level  in  most  Air  Force  develop- 
ments to  find  an  individual  with  authority  over  both  users  and  developers, 
the  emphasis  is  placed  upon  avoiding  being  forced  to  make  such  an  appeal. 
The  developers  prefer  to  rely  on  informal  authority  and  good  relationships. 

Analysis  of  System  Attributes 

One  of  the  objectives  of  this  study  was  to  determine  if  and  how  the 
important  factors  vary  across  a number  of  system  attributes.  These 
attributes  were  measured  by  the  questions  in  Part  B of  the  survey.  Three 
of  the  most  Important  attributes  in  that  part  concerned  the  Success  (8), 
Size  (9)  and  Technical  Difficulty  (13)  of  the  project.  The  responses  to 
each  of  these  questions  were  approximately  normally  distributed,  although 
there  were  relatively  too  many  "very  large"  and  "unsuccessful"  systems 
for  a normal  distribution.  The  mean  rating  for  Success  (9)  was 
"successful"  (3.96),  for  Size  "fairly  large"  (3.91)  and  for  Technical 
Difficulty  "difficult"  (2.77).  As  might  be  expected,  there  was  a nega- 
tive correlation  between  both  Size  and  Technical  Difficulty  and  System 
Success.  This  is  discussed  in  the  section  on  the  "Predictive  Modeling 
Efforts." 

One  advantage  to  the  design  of  Part  C of  the  survey  is  that  it 
permits  analysis  of  each  of  the  variables  from  three  perspectives:  how 
Important  the  variable  was  treated  (Actual),  how  Important  it  should 
have  been  treated  (Ideal),  and  the  difference  between  the  two  (Delta). 
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Each  of  these  perspectives  was  taken  while  analysing  the  variables 
across  the  range  of  Systen  Success.  As  would  be  expected,  few  of  the 
Ideal  ratings  vary  significantly  with  Success.  It  was  interesting  to 
note  however  that  three  variables.  Use  of  a Prototype,  Integration  of 
Systems  and  Contractor  Sespons lveness  (questions  30,  44  and  32)  were 
significantly  more  important  for  unsuccessful  systems  than  for  those 
systems  on  the  remainder  of  the  range  of  Success,  This  Is  a possible 
indication  that  these  variables  might  be  causes  of  failure  when  not 
applied  where  needed. 

A total  of  29  of  the  variables  for  the  Actual  ratings  and  19  of  the 
Delta  ratings  vary  significantly  with  System  Success.  These  were  used 
as  variables  in  the  regression  models  discussed  in  the  last  section  of 
this  chapter.  Of  the  variables  that  vary  significantly,  all  of  the 
Deltas  and  all  but  one  of  the  Actuals  correlate  positively.  That  is, 
as  System  Success  increases,  the  Actual  ratings  Increase  and  the  Delta 
ratings  become  more  positive  (Deltas  are  Actual  ratings  less  Ideal 
ratings).  The  question  with  the  negative  correlation  was  52,  Contractor 
Responsiveness.  This  was  due,  in  ay  opinion,  to  the  confounding  effects 
Sire,  Difficulty  and  Contractual  Involvement  (question  14)  and  not  from 
a fundamentally  negative  relationship.  This  is  discussed  more  fully 
later  in  this  section. 

The  Ideal  variables  which  vary  significantly  with  Size  and  Technical 
Difficulty  are  shown  in  Table  VIII.  That  table  Includes  only  those 
variables  which  also  display  some  organized  pattern  across  the  range  of 
the  two  variables,  be  that  pattern  linear,  curvilinear  or  two-tiered  as 
indicated  in  the  notes  at  the  bottom  of  the  table.  Of  the  variables 
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Table  VIII 

Ideal  Variables  Varying  Significantly  with  System 
Size  or  Technical  Difficulty 


System  Size 

Technical  Difficulty 

High 

Low 

High 

Low 

Variable  Name  (Nr) 

Mean 

Mean 

Sig* 

Mean 

Mean 

Sig* 

Design  Expert  (21) 

4.25 

3.78 

.024 

4.28 

3.90 

.024 

Reviews  (22) 

4.10 

3.40 

.000 

4.19 

3.47 

.015 

Modularization  (23) 

3.70 

3.33 

.005 

3.61 

2.86 

***  .031 

Automated  Tools  (27) 

2.69 

2.24 

.000 

2.78 

2.20 

***  .003 

Prototype  (30) 

3.52 

2.90 

.000 

3.50 

2.62 

.007 

Top  Mgt  Supp  (33) 

4.25 

3.96 

kk 

.000 

4.37 

3.50 

.000 

Design  Staff  (34) 

4.12 

4.00 

irk 

.002 

N.S. 

Str  Proj  Org  (35) 

3.78 

3.15 

kirk 

.018 

4.00 

3.21 

.024 

Ed,  Thg  Users  (37) 

3.83 

3.48 

.010 

3.97 

3.20 

.008 

Career  Mgt  Des  (39) 

3.76 

2.64 

.000 

3.66 

2.80 

.000 

Project  Time  (40) 

3.87 

3.09 

.003 

N.S. 

Proj  Mgr  Auth  (43) 

4.28 

2.41 

.000 

N.S. 

Integration  (44) 

3.96 

3.13 

.000 

4.09 

2.55 

.000 

Proj  Mgr  Ed,  Exp  (46) 

4.31 

3.04 

.000 

4.31 

3.63 

.003 

Proj  Criteria  (50) 

3.93 

3.52 

irk 

.047 

4.22 

3.03 

.005 

Stab  Reqmts  (31) 

4.04 

3.65 

kirk 

.002 

N.S. 

Abil  to  Track  (53) 

3.72 

3.23 

.000 

N.S. 

Justif  of  Cost  (55) 

3.85 

3.30 

.001 

4.06 

3.41 

.018 

Change  Control  (56) 

4.16 

3.72 

.009 

N.S. 

*N.S.  Indicates  not  significant. 

**Non-l inear , middle  values  rated  lowest. 

***Varies  only  at  the  low  or  high  end  of  the  scale. 


shown,  only  four  (21,  39,  43,  and  46)  display  a strongly  linear  trend 
when  viewed  across  system  Sise  and  only  two  (30  and  33)  across  the  range 
of  Technical  Difficulty.  The  high  and  low  means  are  the  mean  of  the 
dependent  variables  for  the  high  and  low  values  of  the  Independent 
variable  (Size  or  Difficulty).  Interestingly,  a combined  Size  Plus 
Difficulty  variable  (recoded  from  the  maximum  possible  11  categories  to 
only  6 categories)  did  a better  Job  of  discriminating  for  10  of  the 
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variables  (23,  27,  30,  33,  35,  37,  44,  50,  52,  and  55)  and  a Size  Times 
Difficulty  variable  (similarly  recoded)  a better  job  of  discriminating 
groups  for  Designer  Expertise  (21). 


A nuaber  of  the 


ideal  variables  vary  significantly  across  the  range 


of  Contractor  Involvement  (question  14)  with  the  clearest  difference 


occurring  between  answers  B and  C ("Contractor  provided  hardware  only" 
and  "Air  Force  and  contractor  each  developed  portions  of  the  operating 
system(s)  . . .").  The  ideal  ratings  were  reexamined  using  question  14 
to  split  the  sample  into  two  groups  called  "Weakly  Contracted" 

(consisting  of  responses  A and  B)  and  "Strongly  Contracted"  (responses  C 
through  F).  The  t»test  of  significance  for  differences  between  means  was 
applied  and  the  16  variables  varying  significantly  are  shown  in  Table  IX. 


Table  IX 

Ideal  Variables  Varying  Significantly  Between 
Weakly  and  Strongly  Contracted  Efforts 


Variable  Name  (Nr) 

Mean  for 
Strongly 
Contracted 

Mean  for 
Weakly 
Contracted 

Slg 

Development  Plan  (20) 

4.21 

3.96 

.011 

Reviews  (22) 

3.99 

3.69 

.004 

Modularization  (23) 

3.63 

3.37 

.047 

Automated  Tools  (27) 

2.39 

2.12 

.035 

Designer  Att  (28) 

3.88 

4.12 

.013 

Prototype  (30) 

3.20 

2.65 

.001 

Top  Mgt  Supp  (33) 

4.12 

3.84 

.011 

Ed,  Tng  Users  (37) 

3.82 

3.50 

.008 

Career  Mgt  Des  (39) 

3.61 

3.17 

.001 

Project  Time  (40) 

3.82 

3.57 

.009 

Integration  (44) 

3.63 

3.23 

.008 

Contractor  Resp  (52) 

3.61 

2.17 

.000 

Abil  to  Track  (53) 

3.65 

3.23 

.000 

Justlf  of  Cost  (55) 

3.63 

3.38 

.040 

Syst  Perf  Critr  (59) 

4.06 

3.86 

.037 

Development  Approach  (29) 

4.06 

3.79 

.015 
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One  difficulty  with  distinguishing  between  strongly  and  weakly  contracted 
efforts  is  the  degree  of  association  between  contracts  and  Size,  Techni- 
cal Difficulty  and  Success.  The  strongly  contracted  efforts  were  some- 
what less  than  "successful"  (4.27)  while  the  weakly  contracted  were 
halfway  between  "successful"  and  "mostly  successful"  (3. 45).  Addition- 
ally, the  strongly  contracted  systems  were  close  to  "large"  (4.67)  while 
the  weakly  contracted  were  closer  to  "medium”  (3.33).  Finally,  the 
strongly  contracted  were  more  than  "difficult"  (3.25)  while  the  weakly 
contracted  were  more  than  "somewhat  difficult"  (2.39).  Given  these 
associations  it  is  difficult  to  say  to  what  extent  the  variables  in 
Table  IX  vary  due  solely  to  Contractor  Involvement.  However,  the  table 
does  present  the  relative  importance  of  Contractor  Respons lveness  (52), 
given  that  the  project  strongly  involved  a contractor.  Given  that 
Contractor  Responsiveness  is  much  more  important  for  strongly  contracted 
efforts,  and  that  contracted  efforts  are  generally  less  successful  (due 
at  least  in  part  to  size  and  technical  difficulty),  the  slight  negative 
correlation  between  Contractor  Responsiveness  and  Success  for  the  total 
sample  is  not  surprising.  Given  that  the  system  is  contracted  to  some 
degree.  Contractor  Responsiveness  has  a slight  positive  correlation 
(zero  order  is  .02). 

The  explanatory  power  of  the  remainder  of  the  Part  B variables  Is 
somewhat  less  than  those  discussed  thus  far.  Concerning  the  Nature  of 
the  Development  as  measured  in  question  10,  it  was  found  that  single 
installations  of  software  and  hardware  were  most  successful  (3.34)  and 
multiple  installations  of  hardware  and  software  least  successful  (4.67). 
Size,  Difficulty  and  Degree  of  Contractor  Involvement  all  increased 
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almost  llnaarly  across  the  four  responses  with  Size  varying  from  2.6  to 
4.8,  Difficulty  from  2. IS  to  3.27  and  Contractor  Involvement  from  1.56  to 
2.78.  Only  two  of  the  Ideal  ratings  varied  significantly  In  light  of 
this  covariance.  Test  Time  (42)  varied  from  a low  of  3.3  for  single 
Installations  of  software  to  3.78  for  single  installations  of  software 
and  hardware.  Information  Needs  (57)  were  by  far  most  Important  for 
single  Installations  of  software  (4.52). 

Analysis  of  question  11,  Nature  of  the  User  Organizatlon(s)  found 
that  projects  Involving  only  a single  user  organization  were  most  suc- 
cessful (3.0),  smallest  (1.73)  and  easiest  (2.37).  Given  this  relation- 
ship, a number  of  variables  were  significantly  less  Important  to  the 
single  user  type  project,  however  all  were  easily  explained  by  the  co- 
variance  of  Success,  Size  and  Difficulty.  None  of  the  factors  varied 
significantly  across  the  other  responses. 

Little  analysis  was  performed  of  question  12,  System  Use,  since  the 
vast  majority  (194)  of  the  responses  Indicated  accounting  type  applica- 
tions. It  was  found  however  that  command  and  control  systems  were  more 
Difficult  and  required  greater  User  Experience  with  Computers. 

It  was  necessary  to  group  some  responses  to  question  15,  Maintenance, 
since  there  were  so  few  responses  indicating  answers  C through  ? (a  total 
of  36).  These  could  be  logically  grouped  as  those  cases  in  which  a 
different  organization  than  the  developers  or  a contractor  performed 
system  maintenance.  Given  this  still  small  cell  count.  System  Success 
was  least  for  that  group,  or  4.75  versus  3.96  and  3.58  for  the  other 
responses.  Contractor  Involvement  was  also  highest  (3.33)  for  that 
group.  None  of  the  factor  ratings  varied  significantly,  other  than 
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Co  the  extent  that  would  be  expected  from  the  variance  in  Contractor 
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Involvement  and  Success. 

It  was  found  that  Success,  Size,  Difficulty  and  Contractor  Involve- 
ment all  varied  quite  significantly  with  the  Use  of  Long-Term  Maintenance 
Warranties  (question  16),  with  the  67  systems  that  used  such  warranties 
the  largest,  most  difficult,  most  contracted  and  least  successful. 

Given  the  variance  with  Size,  Difficulty  and  Contractor  Involvement,  it 
cannot  be  concluded  that  long-term  warranties  make  projects  less  success- 
ful. Ch  the  other  hand,  this  study  found  no  evidence  to  Indicate  that 
such  warranties  make  systems  more  successful. 

Analysis  of  question  17,  Project  Organization,  discovered  a number 
of  interesting  relationships.  Generally  it  was  found  that  projects  with 
no  formal  organization  were  smallest  (2.9)  and  those  organized  into  two 
or  more  independent  teams  under  no  single  manager  (response  E,  with  only 
23  responses)  were  largest  (3.18),  followed  by  two  or  more  teams  under  a 
single  manager  (4.61).  Technical  Difficulty  followed  a similar  pattern. 
Success  was  greatest  for  those  efforts  In  which  there  was  a project 
manager  and  loaned  resources  (3.38),  followed  by  those  with  a formal 
project  organization  (3.90),  two  or  more  teams  under  a single  manager 
(4.01)  and  those  with  no  formal  project  organization  (4.10),  and  finally 
those  with  two  or  more  Independent  teams  under  no  single  manager  (3.13). 
Since  the  efforts  using  a project  organization  were  significantly  larger 
(4.23)  than  those  using  a project  manager  with  loaned  resources  (3.23)  or 
those  with  no  formal  organization  (2.90),  it  can  be  concluded  that  the 
use  of  a formal  project  organization  contributes  materially  to  project 
success  (given  the  negative  relationship  between  Size  and  Success). 
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Similarly*  the  projects  with  no  formal  organization  were  less  success- 
ful  than  the  small  size  would  tend  to  predict*  and  In  general  a case  can 
be  made  for  single  project  leadership. 

Another  point  of  interest  concerning  this  question  is  that  both 
Project  Time  and  Test  Time  (questions  40*  42)  were  observed  to  follow  a 
curvilinear  pattern  across  the  various  organisations*  being  rated  most 
Important  for  those  projects  with  a formal  project  organization  (3.87  and 
3.78)  and  least  important  for  the  projects  at  either  end  of  the  scale. 

One  explanation  for  this  might  be  that  the  tighter  control  associated 
with  a formal  project  organization  allows  greater  emphasis  to  be  placed 
upon  schedule.  The  variance  of  the  other  Part  C variables  was  well  ex- 
plained by  the  variance  in  Size  and  Technical  Difficulty. 

Questions  18  and  19*  Years  Since  the  System  was  Completed  and 
whether  the  System  is  Still  in  Use,  provided  little  explanatory  power 
other  than  the  somewhat  obvious  observation  that  systems  not  still  in 
use  were  less  successful  (since  this  group  included  the  total  failures). 


A number  of  different  directions  were  taken  in  the  course  of  per* 


forming  the  factor  analyses*  the  most  important  of  which  were  the 
analysis  of  the  Actual  ratings  of  the  Part  C variables  for  all  systems* 
and  subsequently  for  strongly  contracted  and  for  weakly  contracted 
systems  (where  partitioning  was  based  on  the  same  criteria  as  was  used 


in  the  preceding  section).  The  Actual  ratings  were  used  rather  than 


the  Ideal  ratings  because  of  the  intent  to  use  the  factors  as  indepen- 
dent variables  in  the  models  for  predicting  System  Success.  Principal 
component  factoring  with  iterations  and  orthogonal  rotation  using  the 


varimax  criterion  was  used  for  all  analyses.  The  factors  were  ortho- 
gonally rotated  In  order  to  derive  maximally  independent.  Interpretable 
factors.  The  varimax  criterion  was  used  to  derive  the  simplest  factors 
by  causing  the  factors  to  load  on  as  few  variables  as  possible  (Ref  41: 
468-485). 

Factor  analysis  was  first  run  with  all  40  variables.  The  general 
procedure  was  than  to  delete  any  variable  not  loading  .30  or  greater  on 
one  or  more  of  the  significant  (eigenvalue  greater  than  1.0)  rotated 
factors  and  rerun  the  analysis.  This  Iterative  procedure  was  followed 
until  only  variables  loading  .30  or  better  on  the  significant  factors 
were  Included.  The  variables  then  not  Included  in  any  of  the  factors 
were  run  separately  (from  the  variables  already  used)  to  determine  if 
any  further  Interpretable  factors  could  be  derived.  The  final  results 
are  shown  in  Table  X. 

The  initial  analysis  for  all  systems  yielded  a total  of  four  sig- 
nificant factors.  The  fourth  loaded  on  the  four  lowest  ranked  variables, 
and  those  variables  were  therefore  discarded  from  the  analysis.  The 
first  factor  consisted  primarily  os  User  Involvement  in  reviews  (31), 
Identification  of  Information  Needs  (57),  Determination  of  the  Objectives 
of  the  User  Organisation  (25)  and  User  Involvement  in  Design  (47).  This 
factor  was  highly  significant  (eigenvalue  9.61  and  explained  51. 4X  of 
variance  in  the  sample)  and  makes  sense  from  the  literature.  The  other 
two  factors  were  tentatively  called  "Control"  and  "Capabilities  of  the 
Project  Organization."  The  final  factor  loadings,  eigenvalues  and  per 
cent  of  variance  explained  are  shown  in  Table  X.  The  previously  excluded 
variables  were  then  Included  in  a separate  analysis.  This  ultimately 
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Table  X 

Summary  of  Results  of  Factor  Analysis  on  Actual  Ratings 


Variable  Name  (Nr) 


User  Obja  (25)  .69 
Users  Review  (31)  .38 
System  Impact  (36)  .51 
Users  Design  (47)  .64 
Info  Needs  (57)  .72 


Factor  Loadings  (Greater  than  .30 


Strong  Weak  Strong  Weak 

All  Contr  Contr  All  Contr  Contr 


User  Support  (58) 
Designer  Att  (28)  .53 

Comm  Ability  (48)  .38 

User  Attitude  (49)  .53 
Designer  Expert(21 

Design  Staff  (34) 

Str  ProJ  Org  (35) 

Proj  Mgr  Exper  (46 
Career  Mgt  Des  (39 
Proj  Itgr  Auth  (43) 

System  Objs  (38) 

Prelim  Design  (41) 
Simp, Log  Des  (54) 

Syst  Perf  Crlt(59) 
Dev'pt  Plan  (20) 

Documentation  (32) 
Reviews  (22) 

Test  Plan  (24) 

Proj  Criteria  (50) 

Abll  to  Track  (53) 

Stab  Reqmts  (51) 

Change  Control  (56 
Justif  of  Cost  (55 


Initial: 

Eigenvalue  9.6  11.0 

Per  Cent  Var  .51  .5 

♦Final : 

Eigenvalue  4.4  6.8 

Per  Cent  Var  .80  .6 

(Relevant  Vars  Only) 

♦See  Text 


.39 

.47 

.50 

.45 

.46 

.31 

.54 

.60 

.48 

.39 

.85 

.72 

.59 

.46 

.62 

.59 

.57 

.71 

.49 

.50 

.51 

.42 

.57 

.43 

.64 

.37 

.57 

.61 

.61  .47 


,50  .59 

,62  .54 

.62 


1.8  2.0  1.7 

.09  .11  .08 


1.4  3.2  1.2 

.23  1.0  .12 


resulted  in  a single  factor,  "Planning  and  Control."  The  use  of  a second 
analysis  explains  why  the  final  per  cent  of  variance  explained  by  Plan- 
ning and  Control  is  100%.  It  was  the  only  factor  in  that  analysis. 

The  analysis  for  strongly  contracted  systems  yielded  similar  results. 
The  first  run  on  this  analysis  disclosed  a total  of  seven  significant 
factors.  The  first  was  again  highly  significant  with  an  eigenvalue  of 
11  and  explained  51%  of  the  variance  in  this  sub-sample.  There  was 
again  a User  Involvement  factor  consisting  primarily  of  Information 
Needs  (57),  User  Support  and  Commitment  (58),  Impact  of  the  System  on 
Users  (36)  ana  System  Objectives  (38).  Thir  was  the  only  factor  in 
which  User  and  System  Objectives  both  appeared,  which  tends  to  discount 
the  argument  mentioned  earlier  that  the  two  may  have  been  erroneously 
associated  by  the  respondents.  Two  of  the  other  initial  factors  re- 
emerged:  "Planning  and  Contr  i."  and  "Capabilities  of  the  Project 
Organization."  The  remaining  factors  were  generally  uninterpretable. 

The  procedure  of  eliminating  variables  was  followed  until  the  final  three 
factors  shown  emerged.  No  second  analysis  was  required  in  this  case, 
and  further  analysis  yielded  unlnterpretable  results. 

The  analysis  of  weakly  contracted  systems  was  similar.  The  Initial 
User  Involvement  factor  (eigenvalue  8.90  and  explained  43.4%  of  the 
variance)  consisted  primarily  of  User  Objectives  (25),  User  Involvement 
in  Reviews  (31)  and  Information  Needs  (57).  For  reasons  unexplainable 
by  this  researcher,  the  Attitude  (28,  49)  and  Communication  (48)  vari- 
ables did  not  enter  or  load  on  this  User  Involvement  factor.  It  may  be 
that  attitude  is  a separate  dimension  for  weakly  contracted  efforts,  but 
such  a factor  failed  to  emerge.  The  final  factors  were  User  Involvement 
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and  Capabilities  of  the  Project  Organization.  An  analysis  of  the 
remaining  variables  disclosed  the  Planning  and  Control  factor  shown. 

This  particular  analysis  demonstrated  the  sensitivity  of  factor  analysis 
(given  a relatively  small  number  of  eases,  147)  to  the  variables  Included 
in  the  analysis.  It  was  found  that  when  the  System  Performance  Criteria 
variable  was  inadvertantly  dropped  from  the  variable  set  that  the  User 
Involvement  factor  dropped  to  second  after  Capabilities.  Whichever 
variable  set  was  used,  the  first  factor  tended  to  be  quite  significant. 
Given  that  User  Involvement  had  been  by  far  the  most  significant  factor 
until  that  point,  I elected  to  retain  the  System  Performance  Criteria 
even  though  it  had  been  subsequently  included  in  the  Planning  and 
Control  factor. 

Thus  the  factors  which  evolved  were  consistent  across  the  three 
analyses.  It  is  Interesting  to  note  the  comparison  between  these  three 
factors  and  the  four  found  by  the  CSU  researchers.  The  four  factors 
found  in  that  study  were  Planning  and  Control,  Attitude,  Expertise  and 
Involvement  (Ref  17s 14).  No  indication  was  found  of  an  Attitude  factor 
in  this  study.  This  may  be  due  in  part  to  the  fact  that  fewer  attitude 
variables  were  Included  In  the  survey.  Two  variables  used  in  the  CSU 
study.  Management  and  Employee  Resistance  to  Change,  were  rated  12  and 
13  (out  of  14)  on  that  study  and  therefore  not  Included  in  this  survey 
(Ref  17:10).  The  naming  of  the  Expertise  factor  in  the  CSU  study  was 
probably  again,  in  my  opinion,  the  result  of  the  variables  included  in 
the  survey.  The  Capabilities  factor  in  this  study  Involves  the  same 
variables,  plus  a number  of  others  relating  to  the  total  human  resources 
of  the  project  organization.  This  is  particularly  the  case  if  attitudes 
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such  as  good  will  are  considered  an  asset.  The  fourth  factor.  User 
Involvement,  was  the  same  although  It  apparently  emerged  much  more 
strongly  In  the  data  from  this  survey.  As  stated  by  the  CSU  researchers, 
"In  the  early  phases  of  this  study,  interviews  with  the  Colorado  business 
community  had  established  'user  involvement*  as  a very  critical  factor” 
(Kef  17:13).  This  study  has  found  that,  at  least  from  the  factor 
analytic  standpoint.  User  Involvement  is  by  far  the  most  critical  factor. 

Other  factor  analyses  performed  included  an  analysis  of  the  Ideal 
ratings  by  both  (separately)  experienced  and  Inexperienced  respondents, 
the  Delta  variables  and  the  Actual  ratings  for  more  successful  projects. 
Of  these  attempts,  only  the  analysis  of  the  Ideal  ratings  by  the  experi- 
enced respondents  yielded  Interpretable  results.  That  analysis  found 
two  factors  in  separate  increments  of  the  variables  interpreted  as  User 
Involvement  and  Planning  and  Control.  The  User  Involvement  factor 
emerged  first  and  consisted  primarily  of  User  Support  (58),  User 
Attitude  (49),  Information  Needs  (57),  User  Involvement  In  Reviews  (31), 
User  Objectives  (25)  and  Designer  Attitude  (28).  The  Planning  and 
Control  factor  consisted  primarily  of  Periodic  Reviews  (22),  Documen- 
tation (32),  Development  Plan  (20),  Preliminary  Design  (41)  and  Work 
Assignment  (45). 

Predictive  Modeling  Efforts 

As  discussed  in  the  last  chapter,  the  Automatic  Interaction 
Detection  (AID)  algorithm  was  used  both  to  indicate  trends  in  the  data 
and  to  point  out  any  unproductive  paths  planned  a priori  in  the  regres- 
sion modeling  efforts.  At  one  point,  the  AID  models  appeared  to  be 
indicating  that  less  experienced  personnel,  both  in  terms  of  years  and 
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number  of  MIS  efforts,  hold  different  opinions  than  more  experienced 
personnel.  This  split  occurred  at  the  third  or  later  levels,  however, 
and  given  the  small  sample  size  (for  AID)  was  probably  indicative  only 
of  Idlosyncracles  in  some  fairly  small  subset  of  the  data.  Modeling 
efforts  without  the  less  experienced  personnel  failed'  to  yield  more 

fc.  I 

consistent  models.  Similarly  the  exclusion  of  responses  relating  to  the 
ALS  (discussed  further  later)  made  little  change  in  the  model.  The  use 
of  the  AID  algorithm  was  generally  unproductive  for  other  purposes.  The 
sample  size  was  too  small  to  allow  this  analyst  co  draw  any  conclusions 
after  the  third  level,  and  the  relationship  between  the  independent 
variables  and  the  criterion  (success)  were  consistent  with  theory  to 
that  point. 

Multiple  regression,  both  with  and  without  dummy  variables,  was 
used  for  the  remainder  of  the  modeling  effort.  Stepwise  regression  with 
the  algorithm  selecting  on  each  iteration  the  variable  vlth  the  greatest 
contribution  toward  total  explained  variance  in  the  criterion  (Success), 
was  used  for  all  models.  Listwlse  deletion  of  cases  with  missing  data 
was  used  for  all  models.  If  a respondent  failed  to  answer  any  of  the 
questions  used  as  variables  in  the  model,  that  entire  set  of  responses 
was  Ignored  for  that  model.  Because  it  seemed  desirable  to  determine 
if  different  variables  became  important  for  predicting  success  for 
different  levels  of  success,  each  model  was  run  for  at  least  three 
ranges  of  success.  The  numerical  assignment  for  responses  to  success 
used  In  regression  is  reversed  from  that  used  in  the  analyses  presented 
earlier.  Highly  successful  systems  were  assigned  a value  of  7.0  and 
unsuccessful  systems  a value  of  1.0.  All  models  were  run  for  the  full 

* 
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rang*  of  Success.  The  models  for  the  total  sample  (called  all  systems) 
and  strongly  contracted  efforts  were  then  run  for  Success  better  than 
"unsuccessful"  (Response  G)  and  for  Success  better  than  "somewhat 
unsuccessful"  (F).  Since  only  three  weakly  contracted  efforts  were 
rated  "unsuccessful,"  those  models  were  run  for  Success  better  than 
"somewhat  unsuccessful"  (F)  and  for  Success  better  than  "mostly 
successful”  (£). 

Most  of  the  models  for  all  systems  and  for  strongly  contracted 
systems  were  also  checked  with  and  without  responses  identified  as  prob- 
able ALS  responses,  on  the  theory  that  the  A IS  may  have  been  such  a 
unique  and  strong  experience  that  it  would  bias  the  models.  The  criteria 
used  for  identifying  the  ALS  responses  was  any  questionnaire  indicating 
AFLC  as  command,  an  unsuccessful  system,  a very  large  system,  and  Techni- 
cal Difficulty  as  either  very  difficult  or  extremely  difficult.  Nine- 
teen responses  fell  into  this  category  for  the  total  sample,  and  after 
llstvise  deletion,  12  ALS  responses  were  identified  in  the  models  for 
all  systems,  and  11  in  the  contracted  systems  models. 

Models  were  created  using  the  Actual  ratings.  Delta  ratings,  and 
selected  Part  9 variables  as  well  as  factors.  In  cases  where  both  the 
Actual  and  Delta  ratings  for  a variable  appeared  as  potential  variables 
in  a regression,  only  the  stronger  predictor  was  retained  in  the  final 
models.  Where  factor  analysis  factors  were  used  as  variables,  neither 
their  component  Actual  variables  nor  the  associated  Deltas  were  used. 
Further,  other  variables  which  seemed  related  to  the  factors  were  not 
used  in  those  cases  where  there  was  a strong  correlation.  Factors  were 
calculated  for  each  case  from  the  formula 

Factor  1 - FSI(XI-X1)/SD1  ♦ FS2 (X2-X2)/SD2  ♦ ...  FSn(Xn-Xn)/SDn  (1) 
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where  PS1  to  FSn  are  the  factor  score  coefficients  for  each  component 


variable  of  the  factor,  XI  to  Xn  the  values  of  the  variables  for  the 
ease,  and  XI  to  Xn  and  SOI  to  SDn  the  mean  and  standard  deviation  for 
Che  variables.  For  ease  of  presentation,  the  factors  in  this  section 
are  shown  simply  as  Factors  1,2,  and  3,  where  Factor  1 refers  always 
to  User  Involvement,  2 to  Capabilities,  and  3 to  Planning  and  Control. 
In  all  cases  the  factors  were  calculated  based  upon  the  factor  analysis 
performed  for  the  same  sample  being  used  in  Che  model.  That  is,  the 
Factor  1 u«ed  in  the  regression  for  strongly  contracted  efforts  is  the 
Factor  1 which  emerged  in  the  factor  analysis  of  that  sub-sample. 

Selected  Part  B questions  were  included  in  the  regression  models 
as  either  continuous  or  dismay  variables.  Size  and  Technical  Difficulty 
(questions  9 and  13)  were  used  as  continuous  variables,  as  were  two 
combined  measures,  Size  Plus  Difficulty  and  Size  Times  Difficulty. 

Part  B questions  10,  14,  13,  and  17,  Nature  of  the  Development,  Con- 

/ 

tractor  Involvement,  Maintenance  and  Project  Organization  were  used  as 
dummy  variables.  The  notation  used  in  this  section  is: 


Question  Nr 

Response (s) 

Development  2 

10 

B 

Development  3 

10 

C 

Development  4 

10 

D 

Contractor  2 

14 

B 

Contractor  3 

14 

C 

Contractor  4 

14 

D,E,F 

Maintenance  2 

13 

B 

Maintenance  3 

13 

C »0»B, F ,G 

Organization  1 

17 

A 

Organization  2 

17 

B 

Organization  4 

17 

D 

Organization  5 

17 

E 

In  each  ease,  as  is  required  for  the  computation  of  the  regression  model, 
one  response  is  omitted  such  that  all  correlations  are  relative  to  the 


response (s)  not  included  in  the  model.  These  reference  responses  were 
the  development  of  software  only  for  s single  installation  for  the 
Development  variables,  maintenance  by  the  same  Air  Force  personnel  who 
developed  the  system  for  the  Maintenance  variables,  no  contractor  in- 
volved for  the  Contractor  variables,  and  a formal  project  organization 
for  the  Organization  variables.  Two  of  the  variables  use  combined 
responses.  Contractor  4 represents  the  cases  where  a contractor  pro- 
vided at  least  part  of  the  operating  system  and  Maintenance  3 the  cases 
where  maintenance  was  performed  by  a contractor  or  an  Air  Force  organi- 
zation other  than  the  developers. 

Hie  results  of  the  modeling  efforts  are  summarized  in  Table  XI. 

The  general  criteria  followed  in  determining  the  models  presented  was 
that  each  Independent  variable  must  contribute  at  least  1.3%  to  total 
explained  variance  (R2)  and  that  each  variable  must  be  significant  at 
the  .05  level  or  better.  These  criteria  were  extended  in  the  cases 
where  dumy  variables  were  Included  in  the  regression  to  allow  a lesser 
contribution  to  explained  variance  and  significance  somewhat  less  than 
.03  (the  lowest  significance  actually  used  was  .082).  In  all  cases,  the 
adjusted  It  was  cheeked  for  an  Increase  with  the  addition  of  each  inde- 
pendent variable.  All  significances  shown  in  the  tables  are  for  the 
contribution  of  the  variable,  given  that  all  of  the  other  variables  are 
in  the  equation.  As  can  be  seen  from  the  table,  the  best  models  were 
found  for  strongly  contracted  efforts,  and  the  models  for  weakly  con- 
tracted efforts  were  the  least  successful.  Additionally,  the  use  of 
factors  in  lieu  of  raw  variables  improved  the  model  in  only  one  case 
(Factor  1 and  variables,  over  variables  only  in  the  regression  for  all 
systems). 
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Table  XI 

Summary  of  Regression  Analyses 


Per 

Cent  Variance  Explained 

(R2) 

All  Systems: 

For  All 
Success 

Without 

ALS 

Better  than 
Unsuccess 

Better  than 
Somewhat 
Unsuccess 

Variables  only 

.33 

.25 

.25 

.18 

Vars,  Dummy  Vars 

.43 

.37 

.30 

.25 

3 Factors  only 

.19(4) 

.20(4) 

.14(2) 

3 Fac,  Vars,  Dummy  Vars 

.39(2) 

.33(2) 

.24(2) 

.19(2) 

3 Fac,  Vars 

.31(1) 

.23(2) 

.17(2) 

.12(2) 

Factor  1,  Vars, 

Dummy  Vars 

.43 

.35 

.26 

.21 

Factor  1,  Vars 

.35 

.24 

.21 

.14 

Approx  Samp  Size 

264 

252 

230 

210 

Contracted: 

Variables  only 

.42 

.37 

.25 

.14 

Vars,  Dummy  Vars 

.52 

.43 

.37 

.37 

3 Factors  only 

.13(4) 

.17(2) 

.11(2) 

3 Fac,  Vars,  Dummy  Vars 

.41(4) 

.25(2) 

.29(2) 

3 Fac,  Vars 

.34(4) 

.15(2) 

.11(2) 

Face  1,3,  Vars, 

Dummy  Vars 

.44(1) 

.32 

.24(1) 

.26(1) 

Facs  1,3,  Vars 

.35 

.22 

.17(1) 

.11(1) 

Approx  Samp  Size 

118 

107 

91 

78 

Noncontracted: 

For  All 
Success 

Better  than 
Somewhat 
Unsuccess 

Better  than 
Mostly 
Success 

Variables  only 

.24 

.22 

.20 

Vars,  Dummy  Vars 

.28 

(3) 

(3) 

3 Factors  only 

.15(2) 

.15(2) 

.13(1) 

3 Facs,  Vars 

.19 

.20 

.30(2) 

3 Facs,  Vars,  Dummy  Vars 

.25(2) 

.20 

(3) 

Fac  1,  Vars 

.19 

.23 

.30 

Fac  1,  Vars,  Dummy  Vars 

.25 

.26 

(3) 

Approx  Sample  Size 

143 

129 

116 

Notes:  (1)  Factor  3 did  not  enter;  (2)  Factors  2,3,  did  not  enter; 
(3)  No  dummy  vars  entered;  (4)  Factor  2 did  not  enter. 
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Selected  regression  aodels  for  all  systems  are  presented  in  Table 
XII.  From  this  table,  it  is  apparent  that  Size  and  Difficulty  are  very 


important  in  predicting  unsuccessful  efforts,  but  not  nearly  as  important 
once  the  unsuccessful  efforts  are  excluded.  The  reasons  for  the  small 
magnitude  correlation  coefficient  for  Size  Times  Difficulty  is  that  the 
maximum  value  was  30.  Conversely,  the  coefficients  for  the  dummy  vari- 
ables are  large  in  magnitude  because  the  maximum  value  is  1.0.  The  more 
appropriate  measure  of  contribution  is  the  Beta  Weight.  Test  Time  (42) 
and  the  Delta  for  Stability  of  Requirements  (51)  are  also  Important  pre- 
dictors of  unsuccessful  efforts.  As  only  more  successful  efforts  are 
included.  Designer  Attitude  (28)  becomes  relatively  more  Important  than 
User  Objectives  (25).  The  other  variable  which  seems  to  Increase  in 
relative  importance  with  success  is  Preliminary  Design  (41).  When 
Factor  1 is  used  in  lieu  of  raw  variables,  it  enters  second  in  place  of 
User  Objectives  and  Designer  Attitude,  two  of  the  main  components  of  the 
factor.  The  main  effect  of  removing  the  ALS  from  consideration  seems  to 
be  to  decrease  the  importance  of  Size  and  Difficulty  in  predicting 
failure.  The  particular  model  shown  also  demonstrates  the  effect  of 
excluding  the  combined  Size  and  Difficulty  variables.  As  is  shown.  Size 
continues  to  enter  with  a negative  coefficient,  although  it  would  enter 
sooner  vith  greater  significance  if  the  ALS  were  not  excluded.  In  those 
models  in  which  the  ALS  was  excluded  and  the  combined  Size  and  Difficulty 
were  variables  included,  the  Sise  Plus  Difficulty  variable  normally 
entered  second  and  added  about  .05  to  explained  variance  (R*).  When  all 
three  factors  are  regressed  without  any  other  variables,  the  result  lsi 
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Factor  1 

.51 

.000 

.17 

Factor  3 

.28 

.064 

.19 

Factor  2 

.17 

.186 

.20 

Table  XII 

Selected  Regression  Models  for  All  Systems 


All  Success.  Vars.  Dummy 


Vars: 

Variable  Name  (Nr) 

Regr 

Coef 

Beta 

Wt 

Sig 

R2 

Change 

R2 

Size  X Difficulty 

-.04 

-.21 

.000 

.16 

.155 

User  Objs  (25) 

.31 

.20 

.000 

.25 

.096 

Development  2 

1.01 

.28 

.000 

.30 

.054 

Designer  Att  (28) 

.24 

.16 

.005 

.34 

.038 

Delta  Proj  Criteria  (50) 

.20 

.16 

.002 

.37 

.028 

Maintenance  3 

-.75 

-.13 

.008 

.39 

.020 

Test  Time  (42) 

.20 

.12 

.013 

.41 

.015 

Maintenance  2 

.40 

.10 

.043 

.42 

.011 

Development  3 

Constant 

.52 

1.79 

.10 

.047 

.43 

.009 

Better  than  Unsuccess. 

Vars . Dumnv  Vars : 

User  Objs  (25) 

.29 

.24 

.000 

.15 

.149 

Designer  Att  (28) 

.22 

.18 

.005 

.20 

.053 

Development  2 

.59 

.21 

.000 

.25 

.043 

Delta  Proj  Criteria  (50) 

.17 

.16 

.005 

.27 

.026 

Maintenance  3 

-.67 

-.14 

.015 

.29 

.021 

System  Impact  (36) 

Constant 

.16 

2.10 

.12 

.047 

.30 

.012 

All  Success.  Variables  Onlv: 

Size  X Difficulty 

-.06 

-.27 

.000 

.15 

.153 

Designer  Att  (28) 

.27 

.18 

.003 

.25 

.094 

Delta  Stability  Reqmts  (51) 

.18 

.12 

.026 

.28 

.036 

User  Objs  (25) 

.25 

.16 

.009 

.30 

.020 

Tost  Time  (42) 

.19 

.12 

.030 

.32 

.016 

Delta  Proj  Criteria  (50) 
Constant 

.16 

2.70 

.12 

.030 

.33 

.013 

All  Success.  Factor  1. 

Vers.  tXmav  Vars: 

Site  X Difficulty 

-.05 

-.26 

.000 

.16 

.161 

Factor  l 

.66 

.35 

.000 

.28 

.120 

Development  2 

.99 

.28 

.000 

.34 

.055 

Delta  Proj  Criteria  (50) 

.24 

.19 

.002 

.36 

.022 

Maintenance  3 

-.83 

-.15 

.003 

.38 

.022 

Test  Time  (42) 

.23 

.14 

.006 

.39 

.014 

Development  3 

.61 

.12 

.020 

.41 

.013 

Average  Delta 

-.62 

-.20 

.008 

.42 

.008 

Oelta  Stability  Reqmts  (51) 
Constant 

.20 

3.66 

.14 

.016 

.43 

.013 
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Table  XII  (continued) 


Selected  Regression  Models  for  All  Systems 

*All  Success.  Factor  1. 

Valuables: 

Variable  Name  (Nr) 

Regr 

Coef 

Beta 

Wt 

Sig 

R2 

Change 

R2 

Size  X Difficulty 

-.07 

-.33 

.000 

.16 

.161 

Factor  1 

.62 

.33 

.000 

.28 

.121 

Delta  Proj  Criteria  (30) 

.23 

.18 

.004 

.30 

.018 

Test  Time  (42) 

.21 

.13 

.018 

.31 

.015 

Proj  Mgr  Ed,  Exper  (46) 

.19 

.12 

.027 

.32 

.010 

Average  Delta 

-.71 

-.23 

.005 

.33 

.010 

Delta  Stability  Reqmts  (51) 

.23 

.16 

.007 

.35 

.019 

Constant 

3.59 

All  Success.  Variables. 
Etoway  Variables. 
Without  ALS: 


Designer  Att  (28) 

.27 

.18 

.002 

.12 

.120 

Development  2 

.81 

.24 

.000 

.20 

.076 

Delta  Stability  Reqmts  (51) 

.17 

.12 

.024 

.24 

.049 

Contractor  4 

-.48 

-.10 

.069 

.28 

.032 

Maintenance  2 

.45 

.12 

.026 

.30 

.024 

Delta  Proj  Criteria  (50) 

.15 

.12 

.029 

.32 

.021 

User  Objs  (25) 

.22 

.15 

.014 

.34 

.015 

Maintenance  3 

-.65 

-.12 

.028 

.35 

.013 

Test  Time  (42) 

.21 

.13 

.012 

.36 

.014 

Size  (9) 

-.13 

-.12 

.026 

.37 

.013 

Constant 

2.39 

*It  erne  decided  to  extend  this  model  beyond  Test  Time,  in  spite  of  the 
low  contributions  by  Proj  Mgr  Ed,  Exper  and  Average  Delta  because  of 
the  larger  contribution  by  Delta  Stability  Reqats. 


As  can  be  seen,  of  the  three  factors,  only  Factor  1,  User  Involvement, 
Is  an  important  predictor  of  System  Success,  although  Factor  3 was  sig- 
nificant at  better  than  .03  prior  to  the  entry  of  Factor  2.  The  effect 
of  the  dummy  variables  is  discussed  at  the  end  of  this  section. 
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Selected  regression  models  for  strongly  contracted  efforts  are 
shown  in  Table  XIII.  The  same  general  observations  made  for  the  models 
of  all  systems  apply  equally  well  here.  One  of  the  main  differences  is 
that  User  Objectives  (25)  are  considerably  less  important  for  predicting 
Success  in  strongly  contracted  efforts  and  the  attitude  "side"  of  User 
Involvement,  as  well  as  the  Impact  of  the  System  an  Users  (36)  are  rela- 
tively  more  important.  Test  Time  seems  to  be  considerably  more  important 
in  strongly  contracted  efforts.  Two  additional  predictors  of  failure 
emerge  in  this  set  of  models,  the  Experience  and  Education  of  the  Project 
Manager  (46)  and  Career  Management  of  the  Designers  (39).  The  Delta  for 
Project  Criteria  (50),  although  an  important  predictor  for  all  systems, 
is  considerably  stronger  for  strongly  contracted  efforts.  The  "Average 
Delta"  variable  is  a simple  unweighted  average  of  all  of  the  Delta  vari- 
ables in  a case.  When  it  enters,  it  always  has  a negative  coefficient, 
simply  indicating  that,  on  the  average,  the  respondents  tend  to  over- 
estimate the  magnitude  of  the  Delta  variables.  The  mean  Average  Delta 
was  -.67.  The  results  of  regressing  only  the  three  factors  were: 


Rear  Coef 

Si£ 

si 

Factor  1 

.31 

.112 

.10 

Factor  2 

.36 

.057 

.13 

Factor  3 

.08 

.682 

.13 

Factors  1 and  2 were  significant  at  .000  and  .042  until  Factor  3 entered 
the  model.  In  general  however  User  Involvement  is  least  Important  as  a 
predictor  in  strongly  contracted  efforts  than  for  either  the  total  sample 
or  for  weakly  contracted  efforts. 

Selected  models  for  weakly  contracted  efforts  are  shown  in  Table  XIV. 
As  indicated  earlier,  the  modeling  effort  for  weakly  contracted  systems 
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Table  XIII 

Selected  Regression  Models  for  Strongly  Contracted  Efforts 


All  Success.  Vars. 


Dummv  Vars: 

Variable  Name  (Nr) 

Regr 

Coef 

Beta 

Wt 

Sig 

si 

Change 

R2 

Size  X Difficulty 

-.07 

-.31 

.000 

.21 

.208 

Test  Time  (42) 

.46 

.27 

.000 

.31 

.104 

Delta  Proj  Criteria  (30) 

.27 

.23 

.002 

.37 

.058 

Development  2 

1.42 

.32 

.000 

.44 

.069 

Delta  System  Impact  (36) 

.29 

.17 

.016 

.47 

.029 

Contractor  4 

-.51 

-.13 

.062 

.49 

.017 

Proj  Mgr  Ed,  Exper  (46) 

.22 

.13 

.060 

.50 

.016 

Development  3 

Constant 

.54 

2.62 

.12 

.082 

.52 

.013 

Better  than  Unsuccess. 

Vars.  Dummv  Vars: 

Delta  Proj  Criteria  (50) 

.23 

.24 

.015 

.16 

.158 

Designer  Att  (28) 

.31 

.26 

.007 

.22 

.060 

Development  2 

.75 

.24 

.008 

.27 

.057 

Delta  System  Impact  (36) 

.23 

.17 

.067 

.32 

.044 

Maintenance  3 

-.67 

-.18 

.050 

.34 

.022 

Delta  Stability  Reqmts  (51) 
Constant 

.22 

3.6) 

.18 

.056 

.37 

.027 

All  Success.  Vars  Onlv: 

Sire  X Difficulty 

-.08 

-.40 

.000 

.22 

.220 

Test  Time  (42) 

.38 

.22 

.005 

.32 

.103 

Delta  Proj  Criteria  (50) 

.22 

.19 

.019 

.37 

.051 

Career  Mgt  Design  (39) 

.22 

.16 

.042 

.40 

.025 

Delta  System  Impact  (36) 
Constant 

.24 

3.39 

.15 

.048 

.42 

.021 

All  Success.  Vars  onlv. 

Without  ALS: 

Test  Time  (42) 

.57 

.35 

.000 

.14 

.140 

Size  ♦ Difficulty 

-.24 

-.29 

.001 

.23 

.085 

Delta  System  Impact  (36) 

.40 

.26 

.007 

.28 

.059 

Delta  Stability  Reqmts  (51) 

.25 

.19 

.037 

.32 

.034 

Delta  Proj  Criteria  (50) 

.29 

.25 

.016 

.33 

.015 

Average  Delta 

-.77 

-.25 

.042 

.35 

.016 

Proj  Mgr  Ed,  Exper  (46) 
Constant 

.27 

3.12 

.16 

.050 

.37 

.025 
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Table  XIII  (continued) 

Selected  Regression  Models  for  Strongly  Contracted  Efforts 


♦All  Success.  Factors  1.3. 

Vars.  Dummy  Varss 

Variable  Name  (Nr) 

Regr 

Coef 

Beta 

Wt 

Sig 

R2 

Change 

R2 

Sice  X Difficulty 

-.07 

-.35 

.000 

.22 

.225 

Factor  1 

.48 

.28 

.000 

.31 

.087 

Development  4 

-.55 

-.16 

.072 

.37 

.054 

Career  Mgt  Design  (39) 

.25 

.17 

.020 

.39 

.027 

Maintenance  3 

-.74 

-.16 

.027 

.42 

.026 

Development  2 

.68 

.15 

.064 

.44 

.026 

Constant 

4.17 

Better  than  Unsuccess. 

Factor  1.  Vars.  Dummy  Varss 

Delta  Project  Criteria  (50) 

.29 

.31 

.004 

.15 

.146 

Factor  1 

.29 

.22 

.038 

.20 

.052 

Development  2 

.71 

.23 

.016 

.25 

.049 

Organization  1 

-.86 

-.19 

.041 

.28 

.032 

Maintenance  3 

-.68 

-.17 

.057 

.31 

.030 

Constant 

4.34 

♦Factor  3 did  not  enter.  The 

reason  this  model 

is  presented  rather 

than  Factor  1 plus  vars  and  dummy  vars 

is  that 

Factor 

1 does  not 

enter 

into  the  contracted  all  success  levels 

model  when  the 

component 

varl- 

ables  of  Factor  3 are  Included 

• 

was  least  successful.  One  general  observation  is  that  the  attitudinal 
variables  seem  to  be  of  considerably  greater  importance  for  predicting 
Success  in  weakly  contracted  efforts  than  for  either  of  the  other  groups. 
As  mentioned  in  the  section  on  the  factor  analysis,  the  attitude  vari- 
ables  did  not  enter  the  User  Involvement  factor  in  the  factor  analysis 
for  weakly  contracted  efforts.  The  zero  order  correlation  between  the 
User  Involvement  factor  and  Designer  Attitude  is  .A3  and  .28  for  the 
Delta  for  User  Attitude  (the  stranger  predictor  when  Factor  1 is  used  In 
the  model).  Although  the  first  is  a relatively  high  correlation,  it  is 
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Table  XIV 

Selected  Regression  Models  for  Weakly  Contracted  Efforts 


f 


All  Success.  Factor  1. 


Vars.  Dumnv  Vars: 
Variable  Name  (Nr) 

Regr 

Coef 

Beta 

Wt 

Slg 

R2 

Chang 

Factor  1 

.43 

.28 

.001 

.15 

.155 

Development  2 

.61 

.21 

.005 

.19 

.037 

Designer  Attitude  (28) 

.29 

.21 

.011 

.23 

.034 

Maintenance  2 

Constant 

.50 

3.10 

.15 

.047 

.25 

.022 

All  Success.  Vars. 

IXimmv  Vars: 

User  Objs  (25) 

.34 

.26 

.001 

.15 

.151 

User  Attitude  (49) 

.27 

.19 

.011 

.20 

.045 

Development  2 

.62 

.21 

.003 

.24 

.041 

System  Impact  (36) 

.23 

.17 

.033 

.27 

.028 

Maintenance  2 

Constant 

.46 

1.30 

.14 

.060 

.28 

.018 

Better  than  Mostlv 

Success.  Factor  1. 

Vars.  Dummv  Vars: 

Prelim  Design  (41) 

.25 

.28 

.002 

.09 

.091 

Designer  Attitude  (28) 

.16 

.18 

.049 

.15 

.063 

System  Integration  (44) 

.17 

.25 

.005 

.19 

.038 

Average  Delta 

-.58 

-.30 

.002 

.22 

.032 

Size  (9) 

• e 17 

-.26 

.007 

.25 

.026 

Factor  1 

.23 

.21 

.039 

.28 

.027 

Difficulty  (13) 

Constant 

.18 

3.01 

.19 

.044 

.30 

.027 

not  high  enough  to  say  that  the  tvo  measure  the  same  dimension.  One 
reasonable  hypothesis  may  be  that  there  is  yet  another  dimension  to 
human  behavior  not  found  by  this  study  that  is  Important  to  success  in 
weakly  contracted  efforts. 

It  is  also  interesting  to  note  that,  for  weakly  contracted  efforts, 
2 

the  highest  R was  found  for  Success  better  than  "mostly  successful." 
This  particular  model  strengthens  the  observation  made  earlier  that 
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Preliminary  Design  (41)  is  a predictor  of  Success  for  more  successful 
efforts.  System  Integration  (44)  emerges  as  a predictor  in  this  model* 
although  not  near  as  strongly  as  Preliminary  Design  or  Designer  Attitude 
(28).  This  variable  also  emerged  at  hi^ier  success  levels  for  some  of 
the  models  of  strongly  contracted  systems  not  shown.  System  Size  (9) 
emerges  in  this  model  with  a negative  coefficient*  and  in  general*  Size 
only  entered  the  models  of  the  weakly  contracted  efforts  at  higher  levels 
of  Success*  and  always  with  a negative  coefficient.  Although  Size  did 
not  enter  the  models  of  more  successful  efforts  for  the  other  two  groups 
at  a significant  level,  it  would  eventually  be  Included  with  a negative 
coefficient  if  the  algorithm  was  allowed  to  continue.  Thus  Size  appar- 
ently continues  to  have  a negative  impact,  even  after  the  very  large, 
difficult  systems  have  been  excluded  from  the  data.  This  was  not  the 
case  with  Technical  Difficulty  which  emerged  as  a predictor  of  Success 
(with  a positive  coefficient)  in  this  and  some  of  the  weakly  contracted 
system  models  for  higher  levels  of  Success.  For  the  other  two  groups* 
Difficulty*  when  it  entered,  always  had  a negative  coefficient.  One 
explanation  might  be  that  for  weakly  contracted  efforts,  given  that  the 
effort  is  mostly  successful  or  better,  increased  difficulty  becomes  a 
measure  of  the  sophistication  or  challenge  associated  with  the  system. 

In  general  dummy  variables  were  found  to  add  considerably  to  the 
predictive  quality  of  the  models  for  both  the  total  sample  and  for 
strongly  contracted  systems,  but  not  the  weakly  contracted  systems. 

This  Is  probably  because  there  is  In  general  less  variability  in  the 
wave  in  which  weakly  contracted  systems  are  developed  and  maintained. 

~he  strongest  predictor  among  the  dummy  variables  was  Development  2,  the 
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development  of  software  and  hardware  for  a single  installation,  which 
always  entered  with  a positive  coefficient.  This  was  followed  by  Main* 
tenance  3,  maintenance  performed  by  a contractor  or  organization  other 
than  the  developers,  which  was  negatively  correlated  with  Success  in  all 
cases.  As  was  mentioned  earlier,  this  variable  was  comprised  of  only  36 
cases.  The  third  most  powerful  was  Development  3,  which  entered  posi- 
tively. Contractor  4,  those  cases  in  which  a contractor  provided  at 
least  the  operating  system,  entered  negatively  when  observed,  as  did 
Organization  1,  no  formal  organization.  The  zero  order  correlations  for 
each  group  of  systems  at  all  levels  of  Success  are  shown  in  Table  XV. 
Although  not  conclusive,  this  table  does  provide  some  implications  for 
policies  concerning  project  organization  and  maintenance. 


Table  XV 

Zero  Order  Correlations  for  Dummy  Variables 


Development  2 

.31 

.26 

.18 

3 

.00 

.13 

.00 

4 

-.34 

-.35 

-.09 

Maintenance  2 

.16 

.16 

.19 

3 

-.22 

-.21 

-.16 

♦Contractor  2 

.04 

3 

-.23 

4 

-.19 

-.05 

Organization  1 

-.04 

-.15 

-.05 

2 

.13 

.14 

.02 

4 

-.02 

-.10 

.01 

5 

-.14 

-.08 

.04 

♦Only  Contractor  4 is 

; available 

in  strongly 

contracted  efforts 

(since 

strongly  contracted 

efforts  are  defined  as 

those  contracted  to 

a degree 

at  least  equivalent 

to  3),  and 

no  contractor  dummy  variables  are 

contracted 

nTPwTW 

Concerning  the  distorting  effects,  if  any,  of  the  ALS,  the  choice 
of  a "best"  model  depends  on  two  issues  which  cannot  be  resolved  by  this 
study.  The  first  is  whether  or  not  the  19  cases  identified  as  referring 
to  the  ALS  Included  all  or  at  least  enough  of  the  ALS  responses  to  test 
the  models  adequately  both  with  and  without  the  ALS.  If  the  ALS  was 
properly  identified,  then  the  main  effect  of  removing  that  system  from 
consideration  is  to  make  the  Size  and  Difficulty  variables  somewhat  less 
powerful  predictors.  The  second  issue  becomes,  given  that  the  ALS  was 
adequately  segregated,  to  what  extent  do  the  distributions  of  the 
Success,  Size  and  Difficulty  variables  with  the  ALS  included  reflect 
reality?  Recalling  the  discussion  of  these  variables  earlier  in  this 
chapter,  each  was  approximately  normally  distributed,  however  there  was 
an  excessive  number  of  "very  large"  systems,  as  well  as  an  excessive 
number  of  "extremely  difficult"  systems  and  similarly  too  many 
"unsuccessful"  systems.  In  terms  of  the  number  of  systems  in  the  popu- 
lation, these  distributions  are  very  likely  Inaccurate.  In  terms  of 
both  the  "kinds  of  systems  people  have  mostly  worked  with"  (given  that 
the  larger  the  project,  the  more  people  Involved)  and  in  terms  of  "total 
dollar  investment,"  these  distributions  may  more  closely  reflect  reality. 
Thus  if  the  ALS  was  adequately  segregated,  then  the  choice  of  model 
depends  upon  the  purpose  to  which  it  is  to  be  applied.  If  the  purpose 
is  to  attain  the  greatest  number  of  successful  systems,  then  it  is  best 
to  select  a "without  ALS”  model,  or  a model  for  Success  greater  than 
unsuccessful.  If  the  purpose  is  to  attain  the  greatest  success  per 
dollar  Invested,  then  the  most  appropriate  model  includes  the  ALS  and 
all  levels  of  Success. 
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Concerning  Che  development  of  two  separate  models  for  predicting 
success  and  failure  discussed  at  the  beginning  of  this  chapter,  it  was 
interesting  to  note  that  some  of  the  Interview  results  were  confirmed 
by  the  survey  analysis.  The  "technical  issues,"  Size  (9)  and  Technical 
Difficulty  (13)  did  prove  to  be  strong  predictors  of  failure.  Similarly, 
requirements  definition,  as  it  was  phrased  in  the  discussion  of  the 
interview  results,  and  Stability  of  Requirements  (51),  the  logical  corol- 
lary of  the  Part  C variables,  both  proved  to  be  predictors  of  failure. 
Unlike  the  Interview  results,  the  survey  analysis  found  the  User  Involve- 
ment factors  to  be  of  importance  throughout  the  range  of  success.  This 

may  be  due  in  part  to  the  dependence  of  good  requirements  definition  upon 

* 

user  Involvement.  The  requirements  must  reflect  what  users  need,  as  is 
reflected  by  the  strong  loading  by  the  User  Involvement  factor  on  both 
User  Objectives  (25)  and  Information  Needs  (57).  In  summary  however 
this  study  did  find  some  evidence  of  two  separate  models  at  work.  The 
strongest  predictors  are  classified  as  predicting  "regardless  of  success 
level,"  "of  failures,"  or  "of  success  given  more  successful  efforts"  in 
Table  XVI. 

This  study  was  generally  unable  to  develop  separate  success  and 
failure  models,  however.  That  is,  the  factor  analysis  of  more  successful 
efforts  did  not  yield  lnterpretable  results,  and  despite  the  classifica- 
tion scheme  of  Table  XVI,  the  predictor  variables  (with  the  exception  of 
some  of  the  failure  predictors),  tended  to  act  as  if  on  a continuum.  For 
example,  as  Success  Increased,  Preliminary  Design  became  gradually  more 
Important.  This  does  not  exclude  hovever  the  possibility  that  there  are 
really  two  potential  criterion  variables,  a success  criterion  and  a 
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Table  XVI 

Classification  of  Strongest  Predictor  Variables 


Strongest  Predictor  Variables: 


Regardless  of 
Success  Level 

Factor  1 

Designer  Att  (28) 
User  Objs  (25) 

Proj  Criteria  (50)** 
System  Impact  (36)** 


Of  Failures 

Size  (9)* 

Difficulty  (13)* 

Test  Time  (42) 

Stab  Reqmts  (51)** 
Proj  Mgr  Ed,  Exp  (46) 
Career  Mgt  Oes  (49) 


Of  Success,  Given  More 
Successful  Efforts 

Preliminary  Deslgn(41) 
User  Attitude  (49) 

To  a Lesser  Degree: 

Integration  (44) 

Design  Expert  (21) 

Test  Plan  (24) 

Design  Staff  (34) 


♦Most  powerfully,  a combination  of  Size  X Difficulty  or  Size  ♦ 
Difficulty,  depending  if  the  ALS  was  included. 

**Used  in  the  regression  models  as  either  an  Actual  variable  or  as  a 
Delta  variable,  depending  on  the  model. 


failure  criterion.  These  might  be  normally  distributed  adjacent  to  each 
other  along  some  grand  continuum  with  an  overlap  between  the  two  curves 
such  that  most  systems  were  operating  somewhere  on  both  curves  at  the 
same  time.  This  might  explain  the  excessive  number  of  systems  in  the 
survey  data  at  the  lower  end  of  the  success  scale.  It  would  require  two 
criterion  variables  and  a complete  set  of  "success"  variables  as  well  as 
a set  of  "failure"  variables  to  test  this  idea,  and  these  were  not  avail* 
able  on  this  survey. 

Critical  Factors 

Although  it  is  beyond  the  scope  of  this  study  to  develop  a concep- 
tual model  of  the  MIS  process,  the  Ideal  variables  rated  as  "Very 
Important"  or  higher  by  the  experienced  respondents  (Table  VI,  page  46) 
suggested  the  partial  process  model  shown  in  Fig.  4.  "Steps"  or 
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Fig.  4.  Ideal  Variables  Rated  as  Very  Important  by  Experienced 
Respondents  Shown  as  a Process  Model. 


"activities"  are  shown  in  boxes  in  the  model,  while  attributes  and  items 
to  be  emphasized  in  order  to  be  successful  are  shown  without  boxes.  The 
dashed  lines  are  used,  in  the  case  of  items  to  be  emphasized,  to  Indicate 
the  portion  of  the  project  during  which  the  item  must  be  considered  to 


achieve  success.  The  Impact  of  the  System  on  Users  for  example  should 
be  considered  right  from  Che  outset  and  then  throughout  the  project. 


The  dashed  lines  in  Che  case  of  activities  indicates  that  other  activities 


not  shown  would  be  performed  between  Che  two  activities  shown.  The  first 
three  activities  were  in  fact  worded  as  activities  on  the  survey,  as  in 
"Determination  of  . . or  "Identification  of  . . . 

Perhaps  the  greatest  difference  between  this  (partial)  model  and 
the  models  in  the  literature  is  that  it  does  not  begin  with  a "Require- 
ment." That  activity  was  not  included  in  the  survey  other  than  in  the 
form  of  Stability  of  Requirements.  However  the  first  three  steps  in 
Fig.  4 constitute  one  wav  to  go  about  defining  the  requirement.  In  my 
opinion,  two  of  the  main  problems  of  development  efforts  listed  in  the 
literature  (constantly  changing  requirements  and  not  delivering  what  the 
user  really  needed)  may  be  due  to  not  following  a basically  similar  pro- 
cedure. It  may  or  may  not  be  coincidental  to  this  model  that  those  three 
steps  were  rated  first,  second  and  third  in  the  order  shown.  However 
much  of  the  literature  is  in  agreement  that  the  earlier  that  a factor 
becomes  important  to  success  in  a project,  the  more  critical  it  is  in 
the  sense  that  it  is  fundamental  to  all  that  follows. 

Another  point  of  Interest  concerning  this  model  is  that  it  consists 
almost  entirely  of  what  some  call  the  "front  end"  or  initial  project 
definition  stages  of  a project.  In  the  Interviews  performed  at  AFDSDC, 
a number  of  personnel  mentioned  the  critical  nature  of  the  front  end. 

The  front  end  was  not  discussed  in  that  section  however  because  it  is 
a combination  of  many  other  variables.  But  with  the  exception  of  Change 
Control  and  poaslbly  Test  Plan,  all  of  the  top  13  variables  are  related 
to  the  front  end.  Test  Plan  may  or  may  not  be,  as  there  is  disagreement 
in  the  literature  about  when  test  planning  should  be  performed,  however 
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nearly  all  of  the  authors  agree  that  at  least  some  test  planning  should 
be  done  In  the  initial  stages  of  the  project. 

Fig.  5 adds  the  strongest  predictor  variables  to  the  Ideal  variables 
in  Fig.  4.  The  predictive  modeling  effort  disclosed  that  Size  and 
Difficulty,  (especially  Size  Times  Difficulty),  Stability  of  Require- 
ments, Test  Time,  Impact  of  the  System  on  Users,  Designer  Attitude, 

Career  Management  of  the  Designers,  Criteria  for  Deciding  to  Continue 
the  Project,  Education  and  Experience  of  the  Project  Manager,  User  Atti- 
tude and  Staffing  of  the  Design  Team  were  all  critical  to  system  success. 
It  is  reasonable  to  add  highly  ranked  Ideal  variables  to  predictor  vari- 
ables in  this  conceptual  model  since  an  Ideal  variable  which  is  being 
given  about  the  right  level  of  attention  in  the  sample  will  not  emerge 
as  a predictor  variable,  even  if  it  is  critical.  The  Ideal  variables  in 
a sense  represent  critical  factors  that  have  been  recognized  and  accounted 
for,  while  the  predictor  variables  have  not  been  adequately  treated  in  at 
least  some  subset  of  less  successful  efforts.  User  Support,  Documenta- 
tion and  Change  Control  are  three  variables  that  fall  into  the  category 
of  being  ranked  highly  on  the  ideal  plane  but  not  emerging  as  predictor 
variables.  The  model  continues  to  be  a model  of  only  the  front  end  of  a 
project,  with  the  exception  of  Test  Time. 

Ihe  three  primary  factor  analyses  performed  in  this  study  (for  the 
total  sample,  strongly  and  weakly  contracted  efforts)  each  revealed  the 
same  three  factors.  The  first  factor.  User  Involvement,  was  highly  sig- 
nificant in  all  cases.  The  second  and  third  factors.  Capabilities  of 
the  Project  Organization  and  Planning  and  Control  were  less  (although 
still)  significant.  When  the  User  Involvement  factor  was  used  in  lieu 
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Fig.  5.  Ideal  Variables  and  Strongest  Predictor  Variables  from 
Regression  Models  Shovn  as  a Process  Model. 
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of  component  variables  in  regression  modeling  it  generally  emerged  as  a 
very  strong  predictor  of  success.  The  three  factors  are  shown  in  Fig.  6 
in  place  of  component  attributes,  but  not  in  place  of  component  steps  or 
activities.  This  considerably  simplifies  the  model. 
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Fig.  6.  Process  Model  of  a Project  Using  Factors  in  Place  of 
Component  Variables. 
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Using  only  the  Independent  (versus  component)  variables  from  this 
model,  the  "Critical  Factors  in  the  Development  of  Air  Force  Computerized 
Management  Information  Systems"  evolving  from  this  study  are: 

1.  User  Involvement. 

2.  Size  and  Difficulty. 

3.  Criteria  to  be  Used  in  Deciding  to  Continue  the  Project. 

4.  Capabilities  of  the  Project  Organization. 

5.  Planning  and  Control. 

6.  Test  Time. 

The  component  variables  shown  in  Fig.  5 can  of  course  be  substituted  for 
the  factors  in  the  list,  and  in  general  the  component  variables  when 
used  in  Regression  resulted  in  a more  powerful  model  in  terms  of  per  cent 
of  explained  variance.  However  the  underlying  dimensions,  that  is  the 
factors,  are  of  sufficient  clarity  that  little  Information  is  lost  and 
much  conceptual  clarity  is  gained  by  using  the  factors  In  lieu  of  vari- 
ables in  a list  of  Critical  Factors.  The  front  end  of  a project  may 
also  be  considered  a critical  factor  in  that  all  of  the  other  critical 
factors  (with  the  exception  of  Test  Time)  relate  closely  to  the  front 
end. 

Recalling  that  the  development  of  a conceptual  model  of  the  MIS 
development  process  is  beyond  the  scope  of  this  study,  I would  however 
like  to  propose  a model  of  the  fro.it  end  which  is  consistent  with  my 
data.  This  is  shown  in  Fig.  7.  The  model  consists  of  the  same  five 
first  steps  shown  In  the  models  presented  earlier.  Each  step  is  followed 
by  a Informal  review  and  approval  of  the  outputs  of  the  step.  For 
example,  the  question  to  be  asked  in  review  of  Information  Needs  is  "Do 
the  Information  Needs  specified  meet  the  stated  User  Objectives,  and  if 
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Products  of  the  Review: 

1.  Commitment  by  Users  to  the  Project  as 
Outlined. 

2.  Evaluation  of  Feasibility  (Size,  Difficulty, 
Tine). 

3.  Evaluation  of  the  Impact  of  the  System 
on  Users. 

4.  Criteria  to  be  Used  in  Deciding  to  Continue/ 
Terminate  the  Project. 

3.  Evaluation,  Commitment  of  Resources. 

6.  System  Performance  Criteria. 

7.  Essential  Elements  of  the  Test  Plan. 

8.  Configuration  Management  Plan  (Change  Control). 

9.  Documentation  Standards. 


*RA  Stands  for  Review  and  Approval. 


Fig.  7.  Proposed  Process  Model  of  the  Front  End  of  a MIS  Development. 


not  which  (needs  or  objectives)  should  be  changed?"  The  Development 
Plan  is  followed  by  a formal  Review  involving  top  and  working  level 
managers  from  both  the  user  and  developer  organizations.  The  products 
to  be  approved  in  the  Review  are  shown.  These  products  address  all  of 
the  critical  factors  not  otherwise  shown  in  the  model,  plus  the  System 
Performance  Criteria. 

This  model  formalizes  User  Involvement  in  three  essential  ways. 
First,  requirements  definition  is  considered  as  three  separate  user- 
oriented  activities.  Second,  reviews  involve  the  user.  Third,  the 
products  of  the  formal  Review  represent  a formal  commitment  by  the  user 
to;  what  the  system  is  to  do;  how  well  the  system  must  do  it;  and  how 
well  the  project  team  must  perform  in  order  to  be  successful.  This 
model  does  not  rule  out  long  range  planning  as  a medium  for  overall 
system  development,  but  rather  can  be  used  to  support  and  extend  the 
planning  process.  Nor  does  the  model  rule  out  incremental  development, 
which  may  be  deemed  essential  if  considerations  of  feasibility  or  objec- 
tives, which  are  particularly  difficult  to  specify,  are  encountered. 

The  model  in  itself  does  not  show,  however,  that  User  Involvement  cannot 
be  completely  formalized.  User  Involvement  depends  a great  deal  upon 
Designer  Attitude  toward  Users  and  on  User  Attitude  toward  Designers. 

The  model  also  cannot  guard  against  changes  Imposed  at  a level  higher 
than  the  user  or  unforeseen  occurrences.  But  it  can,  in  my  opinion, 
make  MIS  development  more  successful. 


- - - ...  • .■  - - ■ - d 
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IV.  Conclusions  and  Recommendations  for  Further  Research 


Conclusions 

As  stated  in  Chapter  I,  the  objectives  of  this  study  were: 

1.  To  determine  what  factors  are  most  important  to  the  successful 
development  of  Air  Fores  computerized  management  information 
systems. 

2.  To  determine  if  and  how  the  important  factors  vary  with  a 
number  of  system  and  project  attributes. 

3.  To  attempt  to  develop  a predictive  model  which  may  be  of  some 
use  in  evaluating  and  managing  future  development  efforts. 

Taking  the  second  objective  first,  it  was  found  that  Size,  Diffi- 
culty and  Degree  of  Contractor  Involvement  were  major  determinants  of 
how  a number  of  the  Ideal  variable  ratings  varied  in  importance.  These 
variables  were  shown  in  Tables  VIII  and  IX.  As  was  expected,  nearly  all 
of  the  significant  variance  was  in  a positive  direction.  That  is,  the 
larger,  more  difficult,  more  strongly  contracted  efforts  required  that 
greater  emphasis  be  placed  upon  the  variables  than  did  the  smaller,  less 
difficult,  weakly  contracted  efforts.  The  one  exception  was  that 
Designer  Attitude  was  significantly  more  important  for  weakly  contracted 
efforts. 

In  general  the  most  linear  relationships  with  Size  were  found  to  be 
In  Designer  Expertise,  Career  Management  of  the  Designers,  Authority 
Available  to  the  Project  Manager  and  the  Education  and  Experience  of  the 
Project  Manager.  In  addition,  the  Use  of  Automated  Development  Tools, 
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Development  of  a Prototype,  Top  Management  Support,  Integration  of 
Systems  and  Ability  to  Track  Cost,  Schedule  and  Performance  varied 
significantly  (at  less  than  the  .001  level)  with  Size. 

Development  of  a Prototype  and  Top  Management  Support  shoved  the 
strongest  linear  relationships  with  technical  difficulty.  In  addition. 
Integration  of  Systems  varied  significantly  at  the  .001  level,  although 
the  relationship  was  not  strongly  linear. 

Concerning  the  Degree  of  Contractor  Involvement,  it  was  difficult 
to  say  with  assurance  to  what  degree  the  variance  of  the  Ideal  variables 
was  due  solely  to  Contractor  Involvement.  That  is,  the  strongly  con- 
tracted efforts  also  tended  to  be  the  largest  and  the  most  difficult. 

Six  of  the  variables  which  varied  significantly  at  the  .05  level  for 
Degree  of  Contractor  Involvement  did  not  vary  significantly  with  either 
Size  or  Technical  Difficulty.  These  were  the  Development  Plan,  Designer 
Attitude,  Contractor  Responsiveness,  Justification  of  Cost,  System 
Performance  Criteria  and  the  Development  Approach.  The  remainder  of  the 
Part  B variables  demonstrated  considerably  less  explanatory  power  than 
did  Size,  Technical  Difficulty  and  Degree  of  Contractor  Involvement. 

A number  of  predictive  models  for  system  success  were  developed  In 
pursuit  of  the  second  objective.  Of  these  models,  the  models  for 
strongly  contracted  efforts  demonstrated  the  highest  percentage  of 
explained  variance  (52%  with  dummy  variables  and  42%  without),  followed 
by  the  models  for  all  systems  (43%  with  dummy  variables  and  35%  without) 
and  the  models  for  weak y contracted  efforts  (28%  with  disomy  variables 
and  24%  without). 
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One  of  the  most  Important  findings  of  the  modeling  effort  was  that 
different  variables  become  important  predictors  of  success  when  unsuccess- 
ful efforts  are  Included  in  the  model  than  when  the  unsuccessful  efforts 
are  excluded.  From  this  it  was  concluded  that  Size,  Technical  Difficulty, 
Test  Time,  Stability  of  Requirements,  Education  and  Experience  of  the 
Project  Manager,  and  Career  Management  of  the  Designers  all  tend  to  be 
predictors  of  failure.  That  is,  these  variables  were  strong  predictors 
when  unsuccessful  efforts  were  Included  in  the  sample  used  to  develop 
Che  model,  but  not  when  the  unsuccessful  efforts  were  excluded.  The 
strongest  predictor  of  failure  was  a combined  variable.  Size  Times 
Difficulty.  The  logical  way  for  a manager  to  handle  this  situation  is 
to  first  insure  chat  the  "failure"  variables  are  adequately  satisfied, 
and  then  to  concentrate  on  the  "success"  variables. 

It  was  found  that  Preliminary  Design,  User  Attitude,  and  to  a lesser 
degree  Integration  of  Systems,  Designer  Expertise,  Test  Plan  and  Staffing 
of  the  Design  Team  were  predictors  of  greater  success,  given  that  only 
more  successful  efforts  were  Included  in  the  model.  That  is,  given  that 
the  project  does  not  fail,  these  variables  are  important  to  greater 
success.  The  User  Involvement  factor.  Designer  Attitude,  Determination 
of  the  Objectives  of  the  User  Organization,  Criteria  for  Deciding  to 
Continue  the  Project  and  Impact  of  the  System  on  Users  were  all  strong 
predictors  of  success,  regardless  of  what  success  levels  were  Included 
in  the  model.  Although,  as  discussed  at  length  in  Chapter  III,  the 
existence  of  separate  failure  and  success  predictors  suggests  the  possi- 
bility of  separate  models  for  success  and  failure,  this  study  Is  unable 
to  provide  such  models  due  to  the  predominantly  one-way  (success) 
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orientation  of  the  data.  It  is  also  consistent  with  the  data  to  consider 


the  failure  variables  as  "minimum"  requirements  for  success;  these  vari- 
ables must  be  satisfied  in  order  to  insure  that  the  project  does  not 
fail,  but  do  not  of  themselves  lead  to  success  greater  than  failure. 

It  was  found  that  the  use  of  dummy  variables  in  general  added  con- 
siderably to  the  predictive  power  of  the  models.  Two  of  the  strongest 
dummy  variables  emerged  from  Question  10,  Nature  of  the  Development. 

From  this  question  it  was  found  that  the  most  successful  efforts  tended 
to  be  those  involving  the  development  of  both  hardware  and  software  for 
a single  installation,  and  the  least  successful  efforts  were  those  in- 
volving multiple  installations  of  hardware  and  software.  The  development 
of  software  only  for  multiple  installations  tended  to  be  either  uncorre- 
lated or  slightly  (positively)  correlated  with  success. 

With  respect  to  Question  IS,  Maintenance,  it  was  found  that  the 
systems  for  which  maintenance  was  performed  by  a organization  other  than 
the  developers  or  by  a contractor  were  least  successful.  However  there 
were  too  few  systems  in  this  category  (36)  to  draw  very  strong  conclu- 
sions. Analysis  of  Question  14,  Degree  of  Contractor  Involvement, 
indicated  that  the  more  strongly  contracted  efforts  were  least  success- 
ful. Question  17,  Project  Organization  was  discussed  in  both  the 
"Analysis  of  System  Attributes"  and  the  "Predictive  Modeling  Efforts" 
sections  of  Chapter  III.  In  summary  it  was  found  that  those  projects 
which  were  formally  organized  under  a project  manager  were  most  success- 
ful, and  those  with  two  or  more  Independent  teams  under  no  single  project 
manager  were  least  successful,  followed  by  those  with  no  formal  project 
organization. 
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Figure  8 sumnarizes  the  results  of  the  predictive  modeling  effort 


by  assigning  a number  of  attributes  to  successful  and  unsuccessful 
efforts.  While,  as  shown,  successful  efforts  tend  to  be  the  reverse  of 
unsuccessful  efforts  (plus  a few  variables),  it  is  not  necessary  for  a 
successful  system  to  be  small  and  easy  from  a technical  standpoint. 
Although  both  variables  are  generally  negatively  correlated,  it  seems  to 
be  sufficient  for  success  that  the  effort  not  be  both  large  and  diffi- 
cult. Also,  while  successful  efforts  are  characterized  by  a high  degree 
of  User  Involvement,  including  the  three  variables  shown,  a fourth.  User 
Attitude  toward  Designers,  is  also  a characteristic  of  successful  efforts. 

Three  different  approaches  were  used  in  pursuit  of  the  first  objec- 
tive of  this  study:  "To  determine  what  factors  are  most  important  to 
the  successful  development  of  Air  Force  computerized  management  informa- 
tion systems."  These  were:  analysis  of  the  Ideal  variable  ratings,  the 
predictive  modeling  effort  and  factor  analysis. 


The  means  and  ranks  for  the  30  most  highly  rated  Ideal  variables 
were  shown  in  Table  VI  (page  46).  It  was  found  that  all  of  the  30  vari- 
ables there  shown  were  of  considerable  importance  in  general,  the  lowest 
mean  being  3.46  (by  inexperienced  respondents),  or  about  half  way  between 
"Important"  and  "Very  Important"  on  the  rating  scale  used  in  the  survey. 
It  vas  also  found  in  the  analysis  of  the  ideal  ratings  that  it  did  not 
make  much  difference  whether  the  variables  were  rated  by  experienced 
(in  MIS)  or  inexperienced  respondents.  I was  unable  to  conclude  however 
that  the  respondents  to  this  survey  and  the  CSU  survey  rated  the  11 
common  variables  the  same,  but  the  correlation  was  close  to  significance. 
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Unsuccessful  Efforts  Tend  To: 


1.  Be  very  large  and  very  difficult  technically. 

2.  Be  characterized  by  a low  degree  of  User  Involvement: 

•Inadequate  Determination  of  User  Objectives. 

•Poor  Attitude  by  Designers  toward  Users. 

•Inadequate  evaluation  of  the  Impact  of  the  System  on  Users. 

3.  Be  those  in  which  inadequate  consideration  was  given  to  the 
Criteria  to  be  used  in  Continuing  the  Project. 

4.  Have  inadequate  Test  Time. 

5.  Have  unstable  Requirements. 

6.  Have  less  qualified  Project  Managers. 

7.  Have  Maintenance  performed  by  an  organization  other  than  the 
developers  or  by  a contractor. 

8.  Have  poor  continuity  in  Designers  (Career  Management  of 
the  Designers). 

9.  Be  more  heavily  Contracted. 

10.  Have  no  formal  organization  or  have  two  or  more  Independent  teams 
with  no  single  project  manager. 

Successful  Efforts  Tend  To: 

*THE  REVERSE,  PLUS: 

1.  Emphasize  Preliminary  Design. 

2.  Have  highly  Expert,  Creative  Designers. 

3.  Involve  the  development  of  software  and  hardware  for  a single 
installation  or  software  only  for  multiple  installations. 

4.  Have  a good  Test  Plan. 

3.  Have  good  Staffing  of  the  Design  Team. 

*See  Text. 


Fig.  3.  Attributes  of  Unsuccessful  and  Successful  Development 
Efforts  as  Found  Through  Multiple  Regression. 


The  most  important  Ideal  variables  were  shown  as  a partial  process 
model  of  the  "front  end"  of  a project  in  Fig.  4 (page  81).  The  most 
important  predictor  variables  from  the  predictive  modeling  efforts  were 
added  to  the  model  in  Fig.  3 (page  84).  This  model  continued  to  be  a 
model  of  only  the  front  end  of  a project. 


The  three  main  factor  analyses  performed  each  revealed  the  same 
three  factors  underlying  the  40  actual  variables  in  the  survey.  These 
were:  User  Involvement,  Capabilities  of  the  Project  Organisation,  and 
Planning  and  Control.  These  three  factors  were  shown  in  the  model  in 
place  of  component  variables  in  Fig.  6 (page  83).  This  considerably 
simplified  the  model  shown  in  the  previous  figure.  The  factors  were 
generally  less  powerful  predictors  of  success  than  their  component 
variables.  However,  the  clarity  of  understanding  provided  by  viewing 
only  the  factors  was  such  that  it  was  elected  to  list  only  the  factors 

i 

as  "Critical  Factors,"  with  the  understanding  that  the  component  vari- 
ables  could  be  substituted.  The  "Critical' Factors  in  the  Development 
of  Air  Force  Computerised  Management  Information  Systems"  were  therefore 
listed  as: 

1.  User  Involvement 

2.  Size  and  Difficulty. 

3.  Criteria  to  be  Used  in  Deciding  to  Continue  the  Project. 

4.  Capabilities  of  the  Project  Organisation. 

3.  Planning  and  Control. 

6.  Test  Time. 

Although  it  vas  beyond  the  scope  of  this  study  to  develop  a concep- 
tual model  of  the  MIS  development  process,  the  previously  discussed 
findings  suggested  a process  model  of  the  front  end  of  a project.  Based 
upon  both  the  findings  of  this  study  and  the  literature  search,  such  a 
model  was  proposed  in  Fig.  7 (page  87).  The  greatest  difference  between 
that  model  and  the  models  shown  predominately  in  the  literature  Is  that 
it  separates  "Requirements  Definition"  into  three  steps:  User  Objectives, 


Information  Needs  and  System  Objectives.  The  model  concluded  the  front 
end  with  a formal  review  which  addressed  all  of  the  Critical  Factors 
found  by  this  study. 

Recommendations  for  Further  Research 

As  is  the  case  with  many  research  efforts,  this  effort  has  raised 
many  unanswered  questions  and  discovered  several  potentially  profitable 
avenues  for  further  research.  This  study  was  in  itself  a follow-on  effort 
to  the  study  performed  by  Deane  Carter  at  Colorado  State  University. 

Four  potential  areas  for  research  concern:  further  analysis  of  the  data 
gathered  by  this  study;  the  replication  of  this  study  with  new  variables 
and  a different  sample  group;  Investigation  of  the  possible  existence  of 
separate  success  and  failure  models;  and  application  of  the  survey  design 
to  other  kinds  of  projects. 

Given  the  limited  time  frame  available  to  this  effort,  the  statis- 
tical analysis  techniques  used  were  of  necessity  only  a start.  Poten- 
tially profitable  areas  for  analysis  of  the  existing  data  include  more 
sophisticated  regression  modeling  methods,  methods  for  controlling  for 
the  effects  of  size  and  difficulty,  and  investigation  of  why  weakly 
contracted  efforts  appear  to  be  in  some  ways  fundamentally  different. 

In  regression  modeling,  a number  of  variables  appeared  to  be  exhibiting 
a nonlinear  relationship  with  success.  This  in  itself  partly  explains 
why  different  factors  tended  to  be  predictors  of  success  and  failure. 

The  confounding  effects  of  Size  and  Difficulty  were  an  obstacle  in  many 
of  the  analysis  efforts.  It  was  for  example  hard  to  say  how  the 
important  variables  varied  for  weakly  versus  strongly  contracted  efforts, 
given  that  strongly  contracted  efforts  were  much  larger  and  more  difficult. 
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Potential  techniques  for  controlling  for  the  effects  of  Size  and  Diffi- 
culty aay  include  canonical  correlation  and  discriminant  analysis.  As 
discussed  in  the  section  an  the  "Predictive  Modeling  Effort,"  weakly 
contracted  efforts  seemed  to  exhibit  fundamentally  different  behavior. 

Investigation  is  needed  into  why  this  may  be  the  case. 

Concerning  possible  replication  of  this  study,  the  most  profitable 
areas  seem  to  be  in  the  use  of  new  variables  and  in  the  possibility  of 
performing  an  "objective"  study.  This  researcher  identified  over  150 
potentially  critical  factors  in  the  literature  search.  Of  these,  40 
were  used  as  factors  in  Part  C of  the  survey  and  12  were  used  as  attri- 
butes in  Part  B.  There  are  without  doubt  important,  if  not  critical 
factors  which  need  to  be  discovered.  Utree  variables  which  should  be 
used  in  modified  form  include  Top  Management  Support  and  Involvement  (33), 

Experience  and  Education  of  the  Project  Manager  (46)  and  Strength  of  the 
Project  Organization  (35).  A number  of  respondents  indicated  that  each 
of  the  first  two  should  be  divided  into  two  separate  variables.  Some 
felt,  for  example,  that  top  management  support  was  critical  but  that  top 
management  involvement  was  undesirable.  Given  that  the  abilities  of  the 
project  manager  were  found  to  be  critical,  added  project  manager  vari- 
ables should  be  considered  to  determine  what  specific  abilities  are  most 
important.  Strength  of  the  Project  Organization  was  Intended  to  measure 
the  degree  of  organizational  unity  (as  opposed  to  two  Independent  teams) 
and  organic  control  over  needed  resources  (as  opposed  to  loaned).  A 
number  of  respondents  commented  that  the  question  was  ambiguous  and  the 
number  of  missing  cases  seemed  to  bear  this  out. 

The  other  area  for  replication  concerns  the  possibility  of  an 
"objective"  study.  It  might  be  possible  to  identify  some  specific 
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projects,  rate  the  projects  on  several  objective  scales  for  both  success 
and  attributes,  and  have  several  personnel  that  were  involved  complete 
Part  C only  of  the  survey.  This  approach  could  also  include  the  users, 
at  least  in  a subjective  evaluation  of  success. 

The  possibility  of  separate  success  and  failure  models  was  discussed 
at  same  length  in  Chapter  III.  In  summary,  this  study  found  seme  evi- 
dence to  support  the  idea,  and  further  investigation  is  needed  as  dis- 
cussed in  that  Chapter. 

The  fundamental  design  of  the  survey,  especially  Part  C,  proved  to 
be  of  considerable  value  in  retrospect.  The  pairing  of  Ideal  and  Actual 
ratings  maLe  the  survey  relatively  fast  to  fill  out  (considering  a total 
of  99  questions  and  the  nature  of  the  Judgments  Involved)  but  provided 
considerable  flexibility  in  analysis  through  the  availability  of  Ideal, 
Actual  and  Delta  ratings.  I believe  that  the  same  design,  and  even  a 
maaber  of  the  same  variables,  can  be  profitably  applied  to  virtually  any 
product-oriented  project  organization  in  which  the  developers  are 
specialists  apart  from  the  users.  The  fundamental  communication  problems 
that  motivate  User  Involvement  are  probably  also  present  in  those  acti- 
vities, although  to  a lesser  degree  than  in  management  information 
systems. 
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FOREWORD 


The  purpose  of  the  attached  survey  is  to  determine  what  factors  are 
most  important  to  the  successful  development  of  Air  Force  computerized 
management  information  systems.  This  survey  is  being  given  to  selected 
officer  and  civilian  personnel  performing  duties  related  to  system 
design*  analysis,  programming  or  operations  at  the  AFESDC,  Gunter  AFS, 
AL,  Hq  AFLC  and  AFSC/ASD  at  Wright-Patterson  AFB,  OH,  and  Hq  SAC/AD  at 
Offutt  AFB,  NE.  Since  the  survey  is  concerned  with  management  informa- 


tion systems,  data  base  management  systems  and  operating  systems,  for 
example,  should  be  considered  only  to  the  extent  that  such  systems  are 
developed  to  support  management  information  systems.  The  survey  is 
concerned  with  command  and  control  systems.  Although  only  a portion  of 
our  business,  the  successful  development  of  management  information  systems 
has  often  proved  to  be  very  difficult  in  the  past,  both  within  and  outside 
the  Air  Force.  In  completing  this  survey,  you  may  feel  that  some  of  the 
questions  are  not  really  applicable  to  your  organization  or  background. 
This  is  because  the  survey  is  designed  to  obtain  information  about  many 
different  kinds  of  systems,  ranging  from  extremely  large  systems  involv- 
ing many  bases  and  new  computers,  to  very  small  individualized  systems 
involving  only  a few  programs  on  existing  computers.  One  of  the  purposes 
of  the  survey  is  to  identify  how  the  important  factors  vary  across  this 
wide  range  of  systems.  Thus,  your  experience  regardless  of  the  size  or 
uniqueness  of  the  project.  Is  Important.  Also,  do  not  be  concerned  if 
your  experience  is  limited  to  either  technical  or  managerial  concerns. 

The  survey  takes  your  job  into  account.  Please  be  candid.  Your  complete 
anonymity  is  assured.  No  attempt  will  be  made  to  Identify  specific 
projects  or  individuals.  The  survey  should  take  about  30  minutes  to 
complete.  Mark  your  answers  directly  on  the  survey.  Please  return  the 
completed  survey  in  the  attached,  stamped  envelope  no  later  than 
30  September  1977.  Your  assistance  is  greatly  appreciated. 


Privacy  Statement 


In  accordance  with  paragraph  30,  AFR  12-33,  the  following  informa- 
tion is  provided  as  required  by  the  Privacy  Act  of  1974. 


Authority. 


(1)  5 U.S.C.  301,  Departments 

(2)  10  U.S.C.  80-12,  Secretar 


of  the 


and/or 

Force.  Powers  and 


B.  Principal  purposes.  The  survey  is  being  conducted  to  collect 
information  to  be  used  in  research  aimed  at  illuminating  and  providing 
Inputs  to  the  solution  of  problems  of  Interest  to  the  Air  Force  and/or 
DOD. 


C.  Routine  uses.  The  survey  data  will  be  converted  to  information 
for  use  in  research  of  management  related  problems.  Results  of  the 
research  based  on  the  data  provided  will  be  included  in  written  Master's 
thesis  and  may  also  be  Included  in  published  articles,  reports  or  texts. 
Distribution  of  the  results  of  the  research,  based  on  the  survey  data, 
whether  in  written  form  or  orally  presented,  will  be  unlimited. 

D*  Participation  in  this  survey  is  entirely  voluntary. 

E.  No  adverse  action  of  any  kind  may  be  taken  against  any  indivi- 
dual  who  elects  not  to  participate  in  any  or  all  parts  of  this  survey. 


SURVEY  PART  A 

CIRCLE  THE  LETTER  IN  FRONT  OF  YOUR  RESPONSE 
1.  Indicate  your  current  assignment 

A.  AFQSDC 

B.  AFLC 

C.  AFSC/ASD 

D.  SAC 

E.  Other,  specify  . 


2.  How  much  experience  do  you  have  in  the  computer  systems  career  field? 


A. 

Less 

than 

1 

year. 

B. 

From 

1 

to  2 

years. 

C. 

From 

2 

to 

3 

years. 

D. 

From 

3 

to 

4 

years. 

E. 

From 

4 

to 

6 

years. 

F. 

From 

6 

to 

8 

years . 

G. 

From 

8 

to 

10  years. 

H. 

From 

10  to  13  years. 

I. 

More 

than 

13 

' years. 

§ 

ft 

rt 

is  your  current  grade? 

A. 

GS-9 

B. 

GS-10 

C. 

GS-ll 

D. 

GS-12 

E. 

CS-13 

F. 

GS-14 

G. 

GS-15 

H. 

0-1 

I. 

0.2 

J. 

0-3 

K. 

0-4 

L. 

0-5 

M. 

0-6 
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4.  What  Is  your  current  specialty  series? 

A.  Civilian  33X  series. 

B.  Military  51XX  series. 

C.  Other  (for  example,  CXXXX) , specify 

3.  Select  the  statement  which  best  describes  your  current  Job. 

If  you  have  been  in  this  job  less  than  six  months,  use  your 
last  job  Involving  computers. 

A.  I work  primarily  in  a managerial  capacity  on  existing  systems. 

B.  I work  primarily  in  a managerial  capacity  on  new  systems. 

C.  I work  primarily  In  a technical  capacity  on  existing  systems. 

0.  1 work  primarily  in  a technical  capacity  on  new  systems. 


6.  How  frequently  does  this  Job  bring  you  into  contact  with  user 
personnel? 

A.  Constantly 

B.  Frequently 

C.  Often 

0.  Sometimes 
C.  Rarely 
F.  Almost  never 


L 
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PART  B 


Answer  the  following  questions  with  respect  to  a single  management 
Information  system  development  effort  in  which  you  recently  participated. 
This  system  should  be  sufficiently  completed  for  you  to  make  some 
judgments  about  the  success  and  problems  of  the  effort. 


8.  Rate  the  successfulness  of  the  development  effort,  considering  both 

project  management  and  the  end  product.  Was  the  project,  for 

example,  completed  on  time  at  projected  cost?  Was  the  system,  for 

example,  usable,  easy  to  maintain,  and  what  the  users  really  needed7 

A.  Highly  Successful . greatly  exceeded  most  requirements. 

B.  Very  Successful,  exceeded  many  requirements. 

C.  Moderately  Successful,  exceeded  a few  requirements,  and  met 
all  Important  ones. 

0.  Successful,  satisfied  all  important  requirements. 

E.  Mostly  Successful,  satisfied  most  important  requirements. 

F.  Somewhat  Unsuccessful,  failed  to  satisfy  several  important 
requirements,  but  produced  a basically  usable  system. 

G.  Unsuccessful,  failed  to  satisfy  major  requirements, 
was  unusable,  barely  usable  or  project  terminated. 


9.  Select  the  statement  which  best  describes  the  size  of  the  project. 

A.  Very  Small,  involved  only  a few  people  for  a few  calendar 
months . 

B.  Small,  involved  several  people  for  up  to  a calendar  year. 

C.  Medium,  involved  a number  of  people  for  more  than  a 
calendar  year. 

0.  Fairly  Large.  Involved  quite  a number  of  people  for  more 
than  a calendar  year. 

E.  Large,  involved  many  people  for  two  or  more  calendar  years. 

F.  Very  Large.  Involved  very  many  people  for  much  longer 
than  two  calendar  years. 


10.  The  project  required  the  development  and/or  procurement  of: 

A.  Software  only,  for  a single  Installation. 

B.  Software  only,  for  multiple  installations. 

C.  Software  and  hardware,  for  a single  installation. 

0.  Software  and  hardware,  for  multiple  Installations. 


1 


II.  Th«  system  was  to  be  used  by: 

A.  A single  user  organization. 

B.  More  than  one  user  organization,  each  with  different 
requirements. 

C.  More  than  one  user  organization  with  similar  requiranents. 

D.  Many  user  organizations,  with  standardized  requirements, 
such  as  those  set  by  regulation. 


12.  The  system  was  to  be  used  primarily  for: 

A.  Accounting  type  applications,  for  example,  finance. 
Inventory,  personnel,  status  or  performance  reporting. 

B.  Command  and  control. 

C.  Scientific  research. 

0.  Project  management. 

E.  Other,  specify 


13.  Rate  the  technical  difficulty  of  the  project. 

A.  Not  Difficult,  example,  proven  software/hardware  in 
proven  applications. 

B.  Somewhat  Difficult,  example,  proven  software/hardware 
in  proven  applications,  but  required  some  techniques 
that  the  developers  were  not  experienced  with. 

C.  Difficult,  example,  unproven  software/hardware  or 
unproven  applications. 

D.  Very  Difficult,  example,  unproven  software/hardware  in 
unproven  applications. 

E.  Extremely  Difficult,  example,  unproven  sof tvare/hardvare 
in  unproven  applications  and  required  the  development 

of  substantial  new  technology. 


14.  Select  the  statement  which  best,  describes  the  degree  of 

contractor  involvement  in  the  project. 

A.  No  contractor  involved. 

B.  Contractor  provided  hardware  only. 

C.  Air  Force  and  contractor  each  developed  portions  of  the 
operating  system(s),  and  the  Air  Force  developed  the 
applications  programs. 

D.  Contractor  provided  all  but  applications  programs. 

E.  Contractor  provided  most  of  the  system.  Air  Force 
developed  some  of  the  applications  programs. 

F.  Contractor  provided  the  complete  system.  Air  Force 
managed  the  project. 


IS.  Who  was  to  maintain  the  system? 
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A. 

B. 

C. 

0. 

E. 

F. 

G. 


16.  Was  the  original  contractor  to  perform  system  maintenance  under 
a long  term  warranty  (3  or  more  years)  included  in  the  original 
contract  award? 

A.  Yes 

B.  Mo 

C.  No  contractor  Involved. 


17.  Select  the  statement  which  best  describes  how  the  project 
was  organized. 

A.  No  formal  project  organization. 

B.  A project  manager  with  resources  loaned  as  needed. 

C.  A formal  project  organization  with  a project  manager. 

0.  Two  or  more  independent  teams  under  a project  manager. 

E.  Two  or  more  independent  teams  under  no  single  project 

manager. 


18.  How  long  ago  was  the  project  completed? 

A.  Project  is  still  going  on. 

B.  Less  than  6 months  ago. 

C.  Between  6 months  and  1 year  ago. 

0.  Between  1 and  2 years  ago. 

E.  Between  2 and  3 years  ago. 

P.  Between  3 and  4 years  ago. 

G.  Between  4 and  6 years  ago. 

H.  Between  6 and  8 years  ago. 

1.  More  than  8 years  ago. 


19.  Is  the  system  being  used  now? 

A.  Yes 

B.  No 

C.  1 don't  know. 
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The  same  Air  Force  personnel  that  developed  the  system. 
Different  Air  Force  personnel  in  the  same  Air  Force 
organization. 

A different  Air  Force  organization  than  the  Air  Force 
organization  that  developed  the  system. 

The  Air  Force,  system  developed  by  contractor. 

The  same  contractor  that  developed  the  system. 

A contractor.  Air  Force  developed  the  system. 

Contractor  other  than  the  original  contractor. 


PART  C 


INSTRUCTIONS  FOR  THOSE  WHO  HAVE  PARTICIPATED  IN  NO  MANAGEMENT 
INFORMATION  SYSTEM  DEVELOPMENT  EFFORTS: 

Rat*  each  of  the  factors  beginning  on  the  next  page  on  the  scale 
shown.  Using  your  experience  and  training,  rate  the  Importance  that 
you  think  should  be  given  to  each  factor  for  the  successful  development 
of  management  information  systems  In  general.  You  might,  for  example, 
think  that  the  development  plan  should  be  treated  as  "Critically 
Important"  ("A"  on  the  rating  scale),  but  that  the  abilities  of  the 
designers  should  be  treated  as  only  "Somewhat  Important"  ("D").  If  you 
think  that  a factor  should  not  be  considered  in  such  projects,  rate  it 
as  "Not  Important  or  Not  Applicable"  ("E").  Fill  your  ratings  in  the 
blanks  in  the  right  hand  column  under  "Ideal  Importance.”  Ignore  the 
left  hand  column  under  "Importance  Given  on  This  Project."  That  column 
is  to  be  used  by  those  who  have  participated  in  one  or  more  development 
efforts.  Begin  on  the  next  page. 


INSTRUCTIONS  FOR  THOSE  WHO  HAVE  PARTICIPATED  IN  ONE  OR  MORE 
MANAGEMENT  INFORMATION  SYSTEM  DEVELOPMENT  EFFORTS: 

Rate  each  of  the  factors  beginning  on  the  next  page,  with  respect 
to  the  same  system  effort  used  in  answering  Part  B.  First,  estimate 
how  much  Importance  was  given  to  the  factor  in  the  development  of  the 
system  by  filling  in  the  blank  in  the  left  hand  column  under  "Importance 
Given  on  This  Project."  Then  rate  how  important  the  factor  should  have 
been  considered  in  order  to  make  the  project  more  successful.  Rate  each 
factor  on  the  scale  shown.  You  might,  for  example,  think  that  the 
development  plan  was  treated  as  "Critically  Important"  ("A"  on  the 
rating  scale),  but  that  it  should  have  been  treated  as  only  "Somewhat 
Important"  ("D").  0£,  for  example,  you  might  think  that  the  abilities 

of  the  designers  was  treated  as  "Somewhat  Important,"  but  that  this 
should  have  been  considered  "Critically  Important."  Rate  according  to 
your  particular  system.  You  might  think  that  some  of  the  factors  are 
generally  Important,  but  were  not  to  your  system.  If  so,  rate  those 
factors  as  "Not  Important  or  Not  Applicable."  Rate  factors  as  important 
only  1£  Important  to  your  system.  Beglr  on  the  next  page. 


RATING  SCALE: 

& 1 £ £ £ 

Critically  Very  Important  Somewhat  Not  Important  or 
Important  Important  Important  Not  Applicable 
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Factor  Ratines 

IF  YOU  WERE  INVOLVED  IN  ONE  OR  MORE  PROJECTS,  FILL  OUT  BOTH 
COLUMNS  FOR  YOUR  SPECIFIC  PROJECT. 

IF  YOU  WERE  INVOLVED  IN  JJO  DEVELOPMENT  EFFORTS , FILL  OUT  THE 
RIGHT  COLUMN  ONLY  FOR  PROJECTS  IN  GENERAL. 


Importance 
Given  on 


Ideal 

Impor- 


This  Project  tance 


_20.  Detail,  coordination  and  currency  of  the 
development  plan. 

„21.  Expertise  and  creative  ability  of  the  designers. 

_22.  Periodic  reviews  of  the  project  at  pre- 
determined phase  points. 

_23.  Modularisation  of  programs. 

_24.  Comprehensiveness  of  the  test  plan.  Including 
test  data. 

_25.  Determination  of  the  objectives  of  the  user 
organisation. 

_26.  User  experience  with  computers. 

_27.  Use  of  automated  development  tools,  such  as 
test  data  generators,  logical  simulators. 

_28.  A positive  attitude  by  the  designers  toward 
the  users. 

_29.  Use  of  a disciplined,  sequential  development 
approach . 

_30.  Development  and  test  of  a prototype. 

_31.  Involvement  by  users  in  the  review  process. 

_32.  Comprehensiveness  and  currency  of  documentation. 

_33.  Involvement  and  support  of  top  management. 


A i £ £ £ 

Critically  Very  Important  Somewhat  Not  Important  or 
Important  Important  Important  Not  Applicable 
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Importance 
Given  on 


Ideal 

Impor- 

tance 


Staffing  of  the  design  team. 

Strength  of  the  project  organization. 

Impact  of  the  system  on  users. 

Education  and  retraining  of  operators 
and  users. 

Definition  of  the  objectives  of  the  system. 

Career  management  of  the  designers,  such  as 
avoiding  PCS,  Insuring  continuity. 

Scheduled  time  for  the  project,  excluding  test. 

Thoroughness  of  the  preliminary  design. 

Scheduled  time  for  testing. 

Authority  available  to  the  project  manager. 

Integration  of  Independent  systems. 

Work  assigned  in  identifiable,  controllable 
packages . 

Experience  and  education  of  the  project 
manager. 

Involvement  in  the  system  design  by  users. 

Personal  communication  ability  of  the 
designers. 

A positive  attitude  by  users  toward  designers. 

Criteria  for  deciding  whether  to  continue 
the  project. 


^ . t I • . a 

Critically  Very 
Important  Important 


• • £ 0 £ 

Important  Somewhat  Not  Important  or 
Important  Not  Applicable 
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Importance 

Ideal 

Given  on 

Impor- 

This  Project 

tance 

— — 

51. 

Stability  of  system  requirements. 

— 

52. 

Contractor  responsiveness  to  problems 
and  changes. 

— 

53. 

Ability  to  track  project  cost,  schedule 
and  performance. 

— 

54. 

Simplicity  and  logic  of  the  overall  system 
design. 

55. 

Justification  of  system  cost. 

■ i- 

56. 

Control  and  coordination  of  changes. 

— 

57. 

Identification  of  the  information  needs 
of  users. 

— . 

58. 

Commitment  and  support  by  users. 

■ ■■  — 

59. 

System  performance  criteria. 

60. 

Other,  specify 

61. 


^ e • e 

• e B e e 

» • e e e e 

e • D e • e 

e e e £ 

Critically 

Very 

Important 

Somewhat 

Not  Important  or 

Important 

Important 

Important 

Not  Applicable 

Thank  you  for 

your  time 

and  effort. 

Please  write 

any  comments  in  the 

space  provided  below. 

Personnel  Interviewed  at  AFDSDC.  Gunter  AFS.  AL 


1.  Beerman,  R.  0.,  Maj.  Deputy  Director  Cor  Medical  Systems. 

AFDSDC/SG. 

2.  Chrlstiani,  A.  B.,  Col.  Director  for  Comptroller  Systems. 

APDSDC/AC. 

3.  Fowler*  B.  L. * Col.  Director  for  Software  Development.  (Formerly 
Director  for  Systems  Technology,  replacing  Col  Phythyon).  AFDSDC/SY. 

4.  Gibbons,  B.  I.,  Ill,  Civ.  AFDSDC/XMPB. 

5.  Glaab,  P.  £.,  Lt  Col.  Deputy  Director  for  Systems  Control. 

AFDS DC/SC. 

6.  Haag,  C.  £.,  CMSGT.  AFDSDC/XMPB. 

7.  Hughes,  J.  D.,  Maj.  Deputy  Director  for  Communl cat Ions. 

AFDS DC/ DC  (Det  1,  CCPC/AFCS). 

8.  Kelly,  E.  F.,  Civ.  AFDSDC/XMY. 

9.  Lewkovich,  J. , Civ.  Deputy  Director  for  Comptroller  Systems. 
AFCSDC/AC. 

10.  Lyne,  T.  L. , Col.  Director  for  Programs  and  Resources.  AFDS DC/PR. 

11.  Maytian,  A.  J.,  Jr.,  Civ.  AFDS  DC /DM. 

12.  Neibling,  R. , Civ.  Deputy  Director  for  Logistics  Systems. 

AFDS DC/LG. 

13.  Noblltt,  J.  W. , Lt  Col.  Director  for  Operations  Systems. 

AFDS DC /XO. 

14.  Orton,  S.  L. , Lt  Col.  Deputy  Director,  Office  of  a.'PS  Management. 
AFDS  X/ DM. 

13.  Phillips,  R.  S.,  Col.  Chief,  Office  of  Plans  and  Management. 
AFDSX/XM. 

16.  Phythyon,  B.  C.,  Col.  Director  for  Software  Development.  (Retiring, 
to  be  replaced  by  Col  Fowler).  AFDSDC/SD. 

17.  Ragsdale,  H.  E.,  Lt  Col.  Chief,  Office  of  Data  Processing.  - 
AFDS DC/ AD. 

18.  Reid,  R.  W.,  Lt  Col.  Director  for  Systems  Control.  AFDSK/SC. 
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